CIHealth
<
Single Day: Oct 7▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 51m 13s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth e2e failed n/a 13
e2e (13)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-94cskf" for cluster "cluster-candidate-4-20-94cskf"
Unexpected error:
    <*fmt.wrapError | 0x14001684f520>: 
    failed to create NodePool np-94cskf: failed waiting for nodepool="np-94cskf" for cluster "cluster-candidate-4-20-94cskf" in resourcegroup="rg-candidate-4-20-dx6vntb24jx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ab0c46ad-c2da-4d9c-a097-c7258ebd8294
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ab0c46ad-c2da-4d9c-a097-c7258ebd8294",
      "name": "ab0c46ad-c2da-4d9c-a097-c7258ebd8294",
      "status": "Failed",
      "startTime": "2026-10-08T00:37:45.188343344Z",
      "endTime": "2026-10-08T01:06:45.910047845Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28 to exist\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28 to exist\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-94cskf" for cluster "cluster-candidate-4-20-94cskf"
Unexpected error:
    <*fmt.wrapError | 0x14001684f520>: 
    failed to create NodePool np-94cskf: failed waiting for nodepool="np-94cskf" for cluster "cluster-candidate-4-20-94cskf" in resourcegroup="rg-candidate-4-20-dx6vntb24jx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ab0c46ad-c2da-4d9c-a097-c7258ebd8294
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ab0c46ad-c2da-4d9c-a097-c7258ebd8294",
      "name": "ab0c46ad-c2da-4d9c-a097-c7258ebd8294",
      "status": "Failed",
      "startTime": "2026-10-08T00:37:45.188343344Z",
      "endTime": "2026-10-08T01:06:45.910047845Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28 to exist\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-bpg28 to exist\nMachine q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-dx6vntb24jx5-candidate-94cskf--managed/providers/Microsoft.Compute/virtualMachines/q3z7q8q2v9s9t7g-np-94cskf-hpd2z-c6965 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6dggkt" for cluster "cluster-candidate-4-21-6dggkt"
Unexpected error:
    <*fmt.wrapError | 0x2b3ad6d92ec0>: 
    failed to create NodePool np-6dggkt: failed waiting for nodepool="np-6dggkt" for cluster "cluster-candidate-4-21-6dggkt" in resourcegroup="rg-candidate-4-21-jgv8rv8mnpsq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c878b86a-b2d7-468c-b2a2-4e3f590ed7b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c878b86a-b2d7-468c-b2a2-4e3f590ed7b4",
      "name": "c878b86a-b2d7-468c-b2a2-4e3f590ed7b4",
      "status": "Failed",
      "startTime": "2026-10-08T00:37:43.744141421Z",
      "endTime": "2026-10-08T01:06:45.988967998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x to exist\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x to exist\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6dggkt" for cluster "cluster-candidate-4-21-6dggkt"
Unexpected error:
    <*fmt.wrapError | 0x2b3ad6d92ec0>: 
    failed to create NodePool np-6dggkt: failed waiting for nodepool="np-6dggkt" for cluster "cluster-candidate-4-21-6dggkt" in resourcegroup="rg-candidate-4-21-jgv8rv8mnpsq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c878b86a-b2d7-468c-b2a2-4e3f590ed7b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c878b86a-b2d7-468c-b2a2-4e3f590ed7b4",
      "name": "c878b86a-b2d7-468c-b2a2-4e3f590ed7b4",
      "status": "Failed",
      "startTime": "2026-10-08T00:37:43.744141421Z",
      "endTime": "2026-10-08T01:06:45.988967998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x to exist\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-7tx5x to exist\nMachine r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-jgv8rv8mnpsq-candidate-6dggkt--managed/providers/Microsoft.Compute/virtualMachines/r3e0a4l3f2r4r2r-np-6dggkt-d8bnk-b5dkg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x25da79d21660>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2wld6d6dj2tk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc4a1e49-a3dd-434a-b2ad-6f784f4be0d8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc4a1e49-a3dd-434a-b2ad-6f784f4be0d8",
      "name": "fc4a1e49-a3dd-434a-b2ad-6f784f4be0d8",
      "status": "Failed",
      "startTime": "2026-10-08T00:08:52.396209625Z",
      "endTime": "2026-10-08T00:38:01.112273549Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5l3c3n7u0v9d0a-np-1-lks62-vc5ds: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-vc5ds to exist\nMachine b5l3c3n7u0v9d0a-np-1-lks62-wcnmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-wcnmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5l3c3n7u0v9d0a-np-1-lks62-vc5ds: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-vc5ds to exist\nMachine b5l3c3n7u0v9d0a-np-1-lks62-wcnmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-wcnmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x25da79d21660>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2wld6d6dj2tk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc4a1e49-a3dd-434a-b2ad-6f784f4be0d8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc4a1e49-a3dd-434a-b2ad-6f784f4be0d8",
      "name": "fc4a1e49-a3dd-434a-b2ad-6f784f4be0d8",
      "status": "Failed",
      "startTime": "2026-10-08T00:08:52.396209625Z",
      "endTime": "2026-10-08T00:38:01.112273549Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5l3c3n7u0v9d0a-np-1-lks62-vc5ds: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-vc5ds to exist\nMachine b5l3c3n7u0v9d0a-np-1-lks62-wcnmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-wcnmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5l3c3n7u0v9d0a-np-1-lks62-vc5ds: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-vc5ds to exist\nMachine b5l3c3n7u0v9d0a-np-1-lks62-wcnmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2wld6d6dj2tk--managed/providers/Microsoft.Compute/virtualMachines/b5l3c3n7u0v9d0a-np-1-lks62-wcnmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh"
Unexpected error:
    <*fmt.wrapError | 0x36df90d3b540>: 
    failed to create NodePool np-tlv9nh: failed waiting for nodepool="np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh" in resourcegroup="rg-candidate-4-22-k8bbp9vgvcln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e",
      "name": "5b294c96-939a-4202-a8fa-48f02c92df7e",
      "status": "Failed",
      "startTime": "2026-10-08T00:39:43.973177646Z",
      "endTime": "2026-10-08T01:08:45.917568185Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh"
Unexpected error:
    <*fmt.wrapError | 0x36df90d3b540>: 
    failed to create NodePool np-tlv9nh: failed waiting for nodepool="np-tlv9nh" for cluster "cluster-candidate-4-22-tlv9nh" in resourcegroup="rg-candidate-4-22-k8bbp9vgvcln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b294c96-939a-4202-a8fa-48f02c92df7e",
      "name": "5b294c96-939a-4202-a8fa-48f02c92df7e",
      "status": "Failed",
      "startTime": "2026-10-08T00:39:43.973177646Z",
      "endTime": "2026-10-08T01:08:45.917568185Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-mgfpd to exist\nMachine g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-k8bbp9vgvcln-candidate-tlv9nh--managed/providers/Microsoft.Compute/virtualMachines/g0k7s1c1z6e9t3k-np-tlv9nh-lbkj4-nq27w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
23:33:31 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2c512700e018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bsqjlrm7bfgr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "name": "597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:05.991580079Z",
      "endTime": "2026-10-08T00:36:11.101154752Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2c512700e018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bsqjlrm7bfgr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "name": "597e749b-3cff-43ff-bf7c-19c6478c2f1e",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:05.991580079Z",
      "endTime": "2026-10-08T00:36:11.101154752Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-6rdnb-gdvcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-gdvcq to exist\nMachine cilium-cluster-cilium-np-6rdnb-x7dhr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bsqjlrm7bfgr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-6rdnb-x7dhr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3133a1380440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-dfm6bjrfc7jr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "name": "30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:37.937984296Z",
      "endTime": "2026-10-08T00:36:41.067847774Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-nb26h-jsv7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-nb26h-jsv7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3133a1380440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-dfm6bjrfc7jr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "name": "30ebea1e-66ce-4e21-ae0f-88cedbdc4981",
      "status": "Failed",
      "startTime": "2026-10-08T00:07:37.937984296Z",
      "endTime": "2026-10-08T00:36:41.067847774Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-nb26h-jsv7r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-nb26h-jsv7r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-dfm6bjrfc7jr--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-nb26h-jsv7r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
23:33:31 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x32955c70b520>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f92csqngd5f6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "name": "c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "status": "Failed",
      "startTime": "2026-10-08T01:32:50.695442989Z",
      "endTime": "2026-10-08T02:01:50.773194309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x32955c70b520>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f92csqngd5f6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "name": "c45e69d4-f127-4c8c-84bd-5e3ef22ca821",
      "status": "Failed",
      "startTime": "2026-10-08T01:32:50.695442989Z",
      "endTime": "2026-10-08T02:01:50.773194309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-psf6x to exist\nMachine o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f92csqngd5f6--managed/providers/Microsoft.Compute/virtualMachines/o4r3a3f0l7z2h4b-np-deletable-52qkj-qqxfj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dbzlxp" for cluster "cluster-candidate-4-23-dbzlxp"
Unexpected error:
    <*fmt.wrapError | 0x3262c3fbda60>: 
    failed to create NodePool np-dbzlxp: failed waiting for nodepool="np-dbzlxp" for cluster "cluster-candidate-4-23-dbzlxp" in resourcegroup="rg-candidate-4-23-7mxgwdtr4c9g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cae9b2c8-0b30-4246-ac5e-55bae91ac247
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cae9b2c8-0b30-4246-ac5e-55bae91ac247",
      "name": "cae9b2c8-0b30-4246-ac5e-55bae91ac247",
      "status": "Failed",
      "startTime": "2026-10-08T00:39:22.908278735Z",
      "endTime": "2026-10-08T01:08:25.937295486Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr to exist\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr to exist\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dbzlxp" for cluster "cluster-candidate-4-23-dbzlxp"
Unexpected error:
    <*fmt.wrapError | 0x3262c3fbda60>: 
    failed to create NodePool np-dbzlxp: failed waiting for nodepool="np-dbzlxp" for cluster "cluster-candidate-4-23-dbzlxp" in resourcegroup="rg-candidate-4-23-7mxgwdtr4c9g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cae9b2c8-0b30-4246-ac5e-55bae91ac247
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/cae9b2c8-0b30-4246-ac5e-55bae91ac247",
      "name": "cae9b2c8-0b30-4246-ac5e-55bae91ac247",
      "status": "Failed",
      "startTime": "2026-10-08T00:39:22.908278735Z",
      "endTime": "2026-10-08T01:08:25.937295486Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr to exist\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-5lwsr to exist\nMachine q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-7mxgwdtr4c9g-candidate-dbzlxp--managed/providers/Microsoft.Compute/virtualMachines/q2d3x1j4c1h2j3i-np-dbzlxp-ngqkn-vqfkj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m"
Unexpected error:
    <*fmt.wrapError | 0x2ac110093d00>: 
    failed to create NodePool np-gcl76m: failed waiting for nodepool="np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m" in resourcegroup="rg-candidate-5-0-fvphxw8q5b7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "name": "891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "status": "Failed",
      "startTime": "2026-10-08T00:42:35.095507926Z",
      "endTime": "2026-10-08T01:11:35.931061942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m"
Unexpected error:
    <*fmt.wrapError | 0x2ac110093d00>: 
    failed to create NodePool np-gcl76m: failed waiting for nodepool="np-gcl76m" for cluster "cluster-candidate-5-0-gcl76m" in resourcegroup="rg-candidate-5-0-fvphxw8q5b7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "name": "891d054c-0f91-44b3-8882-6aa82d1f6e9b",
      "status": "Failed",
      "startTime": "2026-10-08T00:42:35.095507926Z",
      "endTime": "2026-10-08T01:11:35.931061942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-gx48z to exist\nMachine i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-fvphxw8q5b7n-candidate-gcl76m--managed/providers/Microsoft.Compute/virtualMachines/i0y0d7o1e9p6l7u-np-gcl76m-jkvcw-srzbd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tjjkwl" for cluster "cluster-candidate-5-1-tjjkwl"
Unexpected error:
    <*fmt.wrapError | 0x26d616a1d020>: 
    failed to create NodePool np-tjjkwl: failed waiting for nodepool="np-tjjkwl" for cluster "cluster-candidate-5-1-tjjkwl" in resourcegroup="rg-candidate-5-1-q8twtdbssn2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8cd836c-e28b-40d3-9309-d2867dbc70cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8cd836c-e28b-40d3-9309-d2867dbc70cf",
      "name": "c8cd836c-e28b-40d3-9309-d2867dbc70cf",
      "status": "Failed",
      "startTime": "2026-10-08T00:43:44.796892922Z",
      "endTime": "2026-10-08T01:12:45.932063897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx to exist\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx to exist\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tjjkwl" for cluster "cluster-candidate-5-1-tjjkwl"
Unexpected error:
    <*fmt.wrapError | 0x26d616a1d020>: 
    failed to create NodePool np-tjjkwl: failed waiting for nodepool="np-tjjkwl" for cluster "cluster-candidate-5-1-tjjkwl" in resourcegroup="rg-candidate-5-1-q8twtdbssn2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8cd836c-e28b-40d3-9309-d2867dbc70cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8cd836c-e28b-40d3-9309-d2867dbc70cf",
      "name": "c8cd836c-e28b-40d3-9309-d2867dbc70cf",
      "status": "Failed",
      "startTime": "2026-10-08T00:43:44.796892922Z",
      "endTime": "2026-10-08T01:12:45.932063897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx to exist\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-558fx to exist\nMachine t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-q8twtdbssn2z-candidate-tjjkwl--managed/providers/Microsoft.Compute/virtualMachines/t0e4z7t7v6o7i1m-np-tjjkwl-bvg4p-7b9jx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
23:33:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x30a375627b20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x30a375627ae0>{
            msg: "failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x30a375627a80>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x30a375627b20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x30a375627ae0>{
            msg: "failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x30a375627a80>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:33:31 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x1e936bd76030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-vttzcxq26rj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f55dcf67-ecfc-43cf-8a1f-da9dc6f83710
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f55dcf67-ecfc-43cf-8a1f-da9dc6f83710",
      "name": "f55dcf67-ecfc-43cf-8a1f-da9dc6f83710",
      "status": "Failed",
      "startTime": "2026-10-08T00:10:25.217518948Z",
      "endTime": "2026-10-08T00:39:31.060799155Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-w7d69-bsjvv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-w7d69-fjx74: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-vttzcxq26rj2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-w7d69-fjx74 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-w7d69-bsjvv: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-w7d69-fjx74: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-vttzcxq26rj2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-w7d69-fjx74 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x1e936bd76030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-vttzcxq26rj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f55dcf67-ecfc-43cf-8a1f-da9dc6f83710
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f55dcf67-ecfc-43cf-8a1f-da9dc6f83710",
      "name": "f55dcf67-ecfc-43cf-8a1f-da9dc6f83710",
      "status": "Failed",
      "startTime": "2026-10-08T00:10:25.217518948Z",
      "endTime": "2026-10-08T00:39:31.060799155Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-w7d69-bsjvv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-w7d69-fjx74: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-vttzcxq26rj2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-w7d69-fjx74 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-w7d69-bsjvv: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-w7d69-fjx74: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-vttzcxq26rj2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-w7d69-fjx74 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
23:33:31 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x99270993880>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x99270993880>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
2h 25m 39s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 canadacentral e2e failed n/a 25
e2e (25)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nwbn42" for cluster "cluster-nightly-4-23-nwbn42"
Unexpected error:
    <*fmt.wrapError | 0x57354bbc760>: 
    failed to create NodePool np-nwbn42: failed waiting for nodepool="np-nwbn42" for cluster "cluster-nightly-4-23-nwbn42" in resourcegroup="rg-nightly-4-23-z9bjz8bcs4gq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67960c4d-8863-47f5-a18c-ab41c5dd77d6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67960c4d-8863-47f5-a18c-ab41c5dd77d6",
      "name": "67960c4d-8863-47f5-a18c-ab41c5dd77d6",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:50.675391535Z",
      "endTime": "2026-10-08T00:12:57.231820143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g to exist\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g to exist\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nwbn42" for cluster "cluster-nightly-4-23-nwbn42"
Unexpected error:
    <*fmt.wrapError | 0x57354bbc760>: 
    failed to create NodePool np-nwbn42: failed waiting for nodepool="np-nwbn42" for cluster "cluster-nightly-4-23-nwbn42" in resourcegroup="rg-nightly-4-23-z9bjz8bcs4gq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67960c4d-8863-47f5-a18c-ab41c5dd77d6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67960c4d-8863-47f5-a18c-ab41c5dd77d6",
      "name": "67960c4d-8863-47f5-a18c-ab41c5dd77d6",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:50.675391535Z",
      "endTime": "2026-10-08T00:12:57.231820143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g to exist\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-5sd9g to exist\nMachine c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-z9bjz8bcs4gq-nightly-nwbn42--managed/providers/Microsoft.Compute/virtualMachines/c4l7y3i6r1j8y8t-np-nwbn42-zmxvs-gqc4s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-hp5brb" for cluster "cluster-nightly-5-1-hp5brb"
Unexpected error:
    <*fmt.wrapError | 0x215296d8f600>: 
    failed to create NodePool np-hp5brb: failed waiting for nodepool="np-hp5brb" for cluster "cluster-nightly-5-1-hp5brb" in resourcegroup="rg-nightly-5-1-wnq2x6mnbbng" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8919e250-0819-41a0-9b76-ed25de5aa0f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8919e250-0819-41a0-9b76-ed25de5aa0f7",
      "name": "8919e250-0819-41a0-9b76-ed25de5aa0f7",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:39.552205268Z",
      "endTime": "2026-10-08T00:12:47.433037977Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2 to exist\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2 to exist\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-hp5brb" for cluster "cluster-nightly-5-1-hp5brb"
Unexpected error:
    <*fmt.wrapError | 0x215296d8f600>: 
    failed to create NodePool np-hp5brb: failed waiting for nodepool="np-hp5brb" for cluster "cluster-nightly-5-1-hp5brb" in resourcegroup="rg-nightly-5-1-wnq2x6mnbbng" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8919e250-0819-41a0-9b76-ed25de5aa0f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8919e250-0819-41a0-9b76-ed25de5aa0f7",
      "name": "8919e250-0819-41a0-9b76-ed25de5aa0f7",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:39.552205268Z",
      "endTime": "2026-10-08T00:12:47.433037977Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2 to exist\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-9dxv2 to exist\nMachine t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-wnq2x6mnbbng-nightly-hp5brb--managed/providers/Microsoft.Compute/virtualMachines/t4s7o0z3w5g1u6t-np-hp5brb-hrqxs-h2vjl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:00:37 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x964c4f74aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-h5xkjmp44fc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09493ff9-94b2-4569-b1ee-760501a8111a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09493ff9-94b2-4569-b1ee-760501a8111a",
      "name": "09493ff9-94b2-4569-b1ee-760501a8111a",
      "status": "Failed",
      "startTime": "2026-10-07T23:14:22.696850288Z",
      "endTime": "2026-10-07T23:43:29.33911056Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk to exist\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk to exist\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x964c4f74aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-h5xkjmp44fc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09493ff9-94b2-4569-b1ee-760501a8111a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/09493ff9-94b2-4569-b1ee-760501a8111a",
      "name": "09493ff9-94b2-4569-b1ee-760501a8111a",
      "status": "Failed",
      "startTime": "2026-10-07T23:14:22.696850288Z",
      "endTime": "2026-10-07T23:43:29.33911056Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk to exist\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-7hjqk to exist\nMachine f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-h5xkjmp44fc8--managed/providers/Microsoft.Compute/virtualMachines/f8s1r9i4d5k7b5r-np-1-p8nkw-mwv7x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1fa024921260>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-8hhqrj" resourcegroup="rg-np-dg-minor-8hhqrj-7f8bnzmkm895": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-8hhqrj-7f8bnzmkm895/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-8hhqrj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1fa024921260>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-8hhqrj" resourcegroup="rg-np-dg-minor-8hhqrj-7f8bnzmkm895": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-8hhqrj-7f8bnzmkm895/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-8hhqrj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x2e46f700980>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tdmf8r24g52m": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-tdmf8r24g52m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x2e46f700980>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tdmf8r24g52m": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-tdmf8r24g52m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x32c6fec798e0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-vbjww8" in resourcegroup="rg-np-skip-minor-vbjww8-9dlhkbhpd9hb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vbjww8-9dlhkbhpd9hb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vbjww8/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x32c6fec798e0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-vbjww8" in resourcegroup="rg-np-skip-minor-vbjww8-9dlhkbhpd9hb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vbjww8-9dlhkbhpd9hb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vbjww8/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x269719739890>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cfvg2swczz5c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x269719739890>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cfvg2swczz5c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2661dd4d00c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-z8pk7tmfp9jj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2661dd4d00c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-z8pk7tmfp9jj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
23:00:37 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x2e5cf51ae018>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2e5cf5120ca0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf5120c80>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4986960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2e5cf52a2040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf52a2020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4b086e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x2e5cf51ae018>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2e5cf5120ca0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf5120c80>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4986960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2e5cf52a2040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf52a2020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4b086e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x26d333bbc300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-ckbnsnb4ttgr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x26d333bbc300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-ckbnsnb4ttgr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x211d1b817380>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gwwz2wxhg86q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x211d1b817380>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gwwz2wxhg86q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x6c1c704d590>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6dmndbpnqjgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x6c1c704d590>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6dmndbpnqjgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-dl424r" to 4.23.0-0.nightly-multi-2026-10-06-030402
Unexpected error:
    <*exported.ResponseError | 0x23e2acc0cbd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-kjs4dvf659k9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-dl424r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-dl424r" to 4.23.0-0.nightly-multi-2026-10-06-030402
Unexpected error:
    <*exported.ResponseError | 0x23e2acc0cbd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-kjs4dvf659k9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-dl424r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x35088dfe27a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-dwmb74mwp62t": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-dwmb74mwp62t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x35088dfe27a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-dwmb74mwp62t": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-dwmb74mwp62t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-wxpqqn" to 5.0.0-0.nightly-multi-2026-10-06-215158
Unexpected error:
    <*exported.ResponseError | 0x3b69cbc993e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-j89cmzvffx9q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-wxpqqn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-wxpqqn" to 5.0.0-0.nightly-multi-2026-10-06-215158
Unexpected error:
    <*exported.ResponseError | 0x3b69cbc993e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-j89cmzvffx9q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-wxpqqn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pz47xj" for cluster "cluster-nightly-5-0-pz47xj"
Unexpected error:
    <*fmt.wrapError | 0x1e380aa507c0>: 
    failed to create NodePool np-pz47xj: failed waiting for nodepool="np-pz47xj" for cluster "cluster-nightly-5-0-pz47xj" in resourcegroup="rg-nightly-5-0-gv8qpwr567vh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90c14dda-0abc-465a-a86c-5836bdbb53b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90c14dda-0abc-465a-a86c-5836bdbb53b3",
      "name": "90c14dda-0abc-465a-a86c-5836bdbb53b3",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:32.739782591Z",
      "endTime": "2026-10-08T00:12:37.114886242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z to exist\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z to exist\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pz47xj" for cluster "cluster-nightly-5-0-pz47xj"
Unexpected error:
    <*fmt.wrapError | 0x1e380aa507c0>: 
    failed to create NodePool np-pz47xj: failed waiting for nodepool="np-pz47xj" for cluster "cluster-nightly-5-0-pz47xj" in resourcegroup="rg-nightly-5-0-gv8qpwr567vh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90c14dda-0abc-465a-a86c-5836bdbb53b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90c14dda-0abc-465a-a86c-5836bdbb53b3",
      "name": "90c14dda-0abc-465a-a86c-5836bdbb53b3",
      "status": "Failed",
      "startTime": "2026-10-07T23:43:32.739782591Z",
      "endTime": "2026-10-08T00:12:37.114886242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z to exist\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-9rb2z to exist\nMachine y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-gv8qpwr567vh-nightly-pz47xj--managed/providers/Microsoft.Compute/virtualMachines/y2y8x5z5q7w5h6q-np-pz47xj-2xd77-dcwqz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-p2kwgg" to 4.21.0-0.nightly-multi-2026-10-02-115050
Unexpected error:
    <*exported.ResponseError | 0xd94e927e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-xqwwvdsh9nb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-p2kwgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-p2kwgg" to 4.21.0-0.nightly-multi-2026-10-02-115050
Unexpected error:
    <*exported.ResponseError | 0xd94e927e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-xqwwvdsh9nb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-p2kwgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-nfn8dt" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0x353129c55a10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-n89cb96vz49d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-nfn8dt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-nfn8dt" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0x353129c55a10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-n89cb96vz49d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-nfn8dt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bmc2d6" for cluster "cluster-nightly-4-22-bmc2d6"
Unexpected error:
    <*fmt.wrapError | 0xe837a83b40>: 
    failed to create NodePool np-bmc2d6: failed waiting for nodepool="np-bmc2d6" for cluster "cluster-nightly-4-22-bmc2d6" in resourcegroup="rg-nightly-4-22-fbkwmv47s96h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/902c112a-a956-49a6-9b25-1aac16e5506a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/902c112a-a956-49a6-9b25-1aac16e5506a",
      "name": "902c112a-a956-49a6-9b25-1aac16e5506a",
      "status": "Failed",
      "startTime": "2026-10-07T23:39:59.543199325Z",
      "endTime": "2026-10-08T00:09:07.262801862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt to exist\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt to exist\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bmc2d6" for cluster "cluster-nightly-4-22-bmc2d6"
Unexpected error:
    <*fmt.wrapError | 0xe837a83b40>: 
    failed to create NodePool np-bmc2d6: failed waiting for nodepool="np-bmc2d6" for cluster "cluster-nightly-4-22-bmc2d6" in resourcegroup="rg-nightly-4-22-fbkwmv47s96h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/902c112a-a956-49a6-9b25-1aac16e5506a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/902c112a-a956-49a6-9b25-1aac16e5506a",
      "name": "902c112a-a956-49a6-9b25-1aac16e5506a",
      "status": "Failed",
      "startTime": "2026-10-07T23:39:59.543199325Z",
      "endTime": "2026-10-08T00:09:07.262801862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt to exist\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-4mrgt to exist\nMachine b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-fbkwmv47s96h-nightly-bmc2d6--managed/providers/Microsoft.Compute/virtualMachines/b3u9y3i7u2b2x2p-np-bmc2d6-2h4tr-cfqsl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:00:37 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1fd50d650030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-chkfwltmpvqk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a08d520-d555-4d1f-b6ab-c391b10d344e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a08d520-d555-4d1f-b6ab-c391b10d344e",
      "name": "5a08d520-d555-4d1f-b6ab-c391b10d344e",
      "status": "Failed",
      "startTime": "2026-10-07T23:14:17.313989392Z",
      "endTime": "2026-10-07T23:43:19.479231434Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-xc7p4-6l5l8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-6l5l8 to exist\nMachine cilium-cluster-cilium-np-xc7p4-pxtxg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-pxtxg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-xc7p4-6l5l8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-6l5l8 to exist\nMachine cilium-cluster-cilium-np-xc7p4-pxtxg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-pxtxg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1fd50d650030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-chkfwltmpvqk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a08d520-d555-4d1f-b6ab-c391b10d344e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a08d520-d555-4d1f-b6ab-c391b10d344e",
      "name": "5a08d520-d555-4d1f-b6ab-c391b10d344e",
      "status": "Failed",
      "startTime": "2026-10-07T23:14:17.313989392Z",
      "endTime": "2026-10-07T23:43:19.479231434Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-xc7p4-6l5l8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-6l5l8 to exist\nMachine cilium-cluster-cilium-np-xc7p4-pxtxg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-pxtxg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-xc7p4-6l5l8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-6l5l8 to exist\nMachine cilium-cluster-cilium-np-xc7p4-pxtxg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-chkfwltmpvqk--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-xc7p4-pxtxg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
23:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-55x2r2" for cluster "cluster-nightly-4-21-55x2r2"
Unexpected error:
    <*fmt.wrapError | 0x4ebb82b1460>: 
    failed to create NodePool np-55x2r2: failed waiting for nodepool="np-55x2r2" for cluster "cluster-nightly-4-21-55x2r2" in resourcegroup="rg-nightly-4-21-frqv2hgwnkrq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f248ec39-1150-437e-a025-eddf5336b1bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f248ec39-1150-437e-a025-eddf5336b1bb",
      "name": "f248ec39-1150-437e-a025-eddf5336b1bb",
      "status": "Failed",
      "startTime": "2026-10-07T23:39:09.674842789Z",
      "endTime": "2026-10-08T00:08:17.294121295Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js to exist\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js to exist\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-55x2r2" for cluster "cluster-nightly-4-21-55x2r2"
Unexpected error:
    <*fmt.wrapError | 0x4ebb82b1460>: 
    failed to create NodePool np-55x2r2: failed waiting for nodepool="np-55x2r2" for cluster "cluster-nightly-4-21-55x2r2" in resourcegroup="rg-nightly-4-21-frqv2hgwnkrq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f248ec39-1150-437e-a025-eddf5336b1bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f248ec39-1150-437e-a025-eddf5336b1bb",
      "name": "f248ec39-1150-437e-a025-eddf5336b1bb",
      "status": "Failed",
      "startTime": "2026-10-07T23:39:09.674842789Z",
      "endTime": "2026-10-08T00:08:17.294121295Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js to exist\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-2h4js to exist\nMachine v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-frqv2hgwnkrq-nightly-55x2r2--managed/providers/Microsoft.Compute/virtualMachines/v3i7l4w7q0g0q1u-np-55x2r2-gggk4-xd7fs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
23:00:37 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1c11c2b2b440>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-npbf7r72xkcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "name": "26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "status": "Failed",
      "startTime": "2026-10-08T00:01:08.520978007Z",
      "endTime": "2026-10-08T00:30:17.207038187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1c11c2b2b440>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-npbf7r72xkcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "name": "26aa56ea-24a6-4dc0-88b5-bb7b84b010bb",
      "status": "Failed",
      "startTime": "2026-10-08T00:01:08.520978007Z",
      "endTime": "2026-10-08T00:30:17.207038187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-g9n8d to exist\nMachine m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-npbf7r72xkcb--managed/providers/Microsoft.Compute/virtualMachines/m1d0t3b5e4r5u1q-np-delete-main-8bjl8-x4tnk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1829881ca3e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7pmxvz" in resourcegroup="rg-np-version-upgrade-7pmxvz-jjjscx2plvvj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pmxvz-jjjscx2plvvj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pmxvz/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1829881ca3e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7pmxvz" in resourcegroup="rg-np-version-upgrade-7pmxvz-jjjscx2plvvj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pmxvz-jjjscx2plvvj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pmxvz/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-8hxnxn" to 4.22.0-0.nightly-multi-2026-10-06-184130
Unexpected error:
    <*exported.ResponseError | 0x354122bf26f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-jt6zr2jjbm9m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-8hxnxn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-8hxnxn" to 4.22.0-0.nightly-multi-2026-10-06-184130
Unexpected error:
    <*exported.ResponseError | 0x354122bf26f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-jt6zr2jjbm9m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-8hxnxn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
23:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf"
Unexpected error:
    <*fmt.wrapError | 0x312a48ea2980>: 
    failed to create NodePool np-zgtwtf: failed starting nodepool creation "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf" in resourcegroup="rg-nightly-4-20-j42q4dvn7t74": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-j42q4dvn7t74/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zgtwtf/nodePools/np-zgtwtf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf"
Unexpected error:
    <*fmt.wrapError | 0x312a48ea2980>: 
    failed to create NodePool np-zgtwtf: failed starting nodepool creation "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf" in resourcegroup="rg-nightly-4-20-j42q4dvn7t74": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-j42q4dvn7t74/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zgtwtf/nodePools/np-zgtwtf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 54m 22s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth e2e failed n/a 11
e2e (11)
client rate limiter Wait returned an error: context deadline exceeded
19:33:35 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x42d11f950c8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2vrv7v7vlpcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d",
      "name": "9a289f01-1747-41f4-af5e-032e7e217e1d",
      "status": "Failed",
      "startTime": "2026-10-07T20:08:33.943284812Z",
      "endTime": "2026-10-07T20:37:40.405868199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x42d11f950c8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2vrv7v7vlpcb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9a289f01-1747-41f4-af5e-032e7e217e1d",
      "name": "9a289f01-1747-41f4-af5e-032e7e217e1d",
      "status": "Failed",
      "startTime": "2026-10-07T20:08:33.943284812Z",
      "endTime": "2026-10-07T20:37:40.405868199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-lrj9w-6k6fg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-6k6fg to exist\nMachine cilium-cluster-cilium-np-lrj9w-hx9gl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-2vrv7v7vlpcb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-lrj9w-hx9gl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tng4gr" for cluster "cluster-candidate-4-20-tng4gr"
Unexpected error:
    <*fmt.wrapError | 0x38fde1c16bc0>: 
    failed to create NodePool np-tng4gr: failed waiting for nodepool="np-tng4gr" for cluster "cluster-candidate-4-20-tng4gr" in resourcegroup="rg-candidate-4-20-p58t2pzhvwgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e7e65054-2290-43e7-8239-944663da4cfa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e7e65054-2290-43e7-8239-944663da4cfa",
      "name": "e7e65054-2290-43e7-8239-944663da4cfa",
      "status": "Failed",
      "startTime": "2026-10-07T20:35:56.19805172Z",
      "endTime": "2026-10-07T21:04:58.851835237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp to exist\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp to exist\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tng4gr" for cluster "cluster-candidate-4-20-tng4gr"
Unexpected error:
    <*fmt.wrapError | 0x38fde1c16bc0>: 
    failed to create NodePool np-tng4gr: failed waiting for nodepool="np-tng4gr" for cluster "cluster-candidate-4-20-tng4gr" in resourcegroup="rg-candidate-4-20-p58t2pzhvwgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e7e65054-2290-43e7-8239-944663da4cfa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e7e65054-2290-43e7-8239-944663da4cfa",
      "name": "e7e65054-2290-43e7-8239-944663da4cfa",
      "status": "Failed",
      "startTime": "2026-10-07T20:35:56.19805172Z",
      "endTime": "2026-10-07T21:04:58.851835237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp to exist\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-55wrp to exist\nMachine x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-p58t2pzhvwgd-candidate-tng4gr--managed/providers/Microsoft.Compute/virtualMachines/x4a6c1x6c1p1p4v-np-tng4gr-mkzdj-vlpx2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xe2ef1777c20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-jtndxjs7xcxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ec092cc-d390-467a-9cfc-6f58b823214c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ec092cc-d390-467a-9cfc-6f58b823214c",
      "name": "8ec092cc-d390-467a-9cfc-6f58b823214c",
      "status": "Failed",
      "startTime": "2026-10-07T20:08:57.296668233Z",
      "endTime": "2026-10-07T20:38:00.537683031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-bnc9s-6hbvx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jtndxjs7xcxx--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bnc9s-6hbvx to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-bnc9s-6hbvx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jtndxjs7xcxx--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bnc9s-6hbvx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xe2ef1777c20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-jtndxjs7xcxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ec092cc-d390-467a-9cfc-6f58b823214c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8ec092cc-d390-467a-9cfc-6f58b823214c",
      "name": "8ec092cc-d390-467a-9cfc-6f58b823214c",
      "status": "Failed",
      "startTime": "2026-10-07T20:08:57.296668233Z",
      "endTime": "2026-10-07T20:38:00.537683031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-bnc9s-6hbvx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jtndxjs7xcxx--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bnc9s-6hbvx to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-bnc9s-6hbvx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jtndxjs7xcxx--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bnc9s-6hbvx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1c37066c9ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-p77gk7tcv5q9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8efd0a6-5b62-424e-9f06-59f6979e8fd6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8efd0a6-5b62-424e-9f06-59f6979e8fd6",
      "name": "c8efd0a6-5b62-424e-9f06-59f6979e8fd6",
      "status": "Failed",
      "startTime": "2026-10-07T20:07:39.49072968Z",
      "endTime": "2026-10-07T20:36:40.369440399Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-68sdx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-68sdx to exist\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-68sdx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-68sdx to exist\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1c37066c9ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-p77gk7tcv5q9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8efd0a6-5b62-424e-9f06-59f6979e8fd6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c8efd0a6-5b62-424e-9f06-59f6979e8fd6",
      "name": "c8efd0a6-5b62-424e-9f06-59f6979e8fd6",
      "status": "Failed",
      "startTime": "2026-10-07T20:07:39.49072968Z",
      "endTime": "2026-10-07T20:36:40.369440399Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-68sdx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-68sdx to exist\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-68sdx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-68sdx to exist\nMachine q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-p77gk7tcv5q9--managed/providers/Microsoft.Compute/virtualMachines/q5l4t9b8y5f2w6p-np-1-l68gq-6pp4h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-9cltms" for cluster "cluster-candidate-4-21-9cltms"
Unexpected error:
    <*fmt.wrapError | 0x17226b681ca0>: 
    failed to create NodePool np-9cltms: failed waiting for nodepool="np-9cltms" for cluster "cluster-candidate-4-21-9cltms" in resourcegroup="rg-candidate-4-21-5wcq5gcwvh45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a18c5c3-1285-4432-9e6e-bd738edcf21c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a18c5c3-1285-4432-9e6e-bd738edcf21c",
      "name": "5a18c5c3-1285-4432-9e6e-bd738edcf21c",
      "status": "Failed",
      "startTime": "2026-10-07T20:36:07.718000912Z",
      "endTime": "2026-10-07T21:05:08.775957747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5 to exist\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5 to exist\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-9cltms" for cluster "cluster-candidate-4-21-9cltms"
Unexpected error:
    <*fmt.wrapError | 0x17226b681ca0>: 
    failed to create NodePool np-9cltms: failed waiting for nodepool="np-9cltms" for cluster "cluster-candidate-4-21-9cltms" in resourcegroup="rg-candidate-4-21-5wcq5gcwvh45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a18c5c3-1285-4432-9e6e-bd738edcf21c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5a18c5c3-1285-4432-9e6e-bd738edcf21c",
      "name": "5a18c5c3-1285-4432-9e6e-bd738edcf21c",
      "status": "Failed",
      "startTime": "2026-10-07T20:36:07.718000912Z",
      "endTime": "2026-10-07T21:05:08.775957747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5 to exist\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-5q8n5 to exist\nMachine m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5wcq5gcwvh45-candidate-9cltms--managed/providers/Microsoft.Compute/virtualMachines/m2a5h3r9k9e3a3i-np-9cltms-rbspc-76sz5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6rlkdh" for cluster "cluster-candidate-4-23-6rlkdh"
Unexpected error:
    <*fmt.wrapError | 0x15014d7b5940>: 
    failed to create NodePool np-6rlkdh: failed waiting for nodepool="np-6rlkdh" for cluster "cluster-candidate-4-23-6rlkdh" in resourcegroup="rg-candidate-4-23-cbh64xfpcppm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30c1fd25-bea7-4668-ba77-9639e04fbe20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30c1fd25-bea7-4668-ba77-9639e04fbe20",
      "name": "30c1fd25-bea7-4668-ba77-9639e04fbe20",
      "status": "Failed",
      "startTime": "2026-10-07T20:41:47.534201418Z",
      "endTime": "2026-10-07T21:10:48.799809829Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9 to exist\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9 to exist\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6rlkdh" for cluster "cluster-candidate-4-23-6rlkdh"
Unexpected error:
    <*fmt.wrapError | 0x15014d7b5940>: 
    failed to create NodePool np-6rlkdh: failed waiting for nodepool="np-6rlkdh" for cluster "cluster-candidate-4-23-6rlkdh" in resourcegroup="rg-candidate-4-23-cbh64xfpcppm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30c1fd25-bea7-4668-ba77-9639e04fbe20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/30c1fd25-bea7-4668-ba77-9639e04fbe20",
      "name": "30c1fd25-bea7-4668-ba77-9639e04fbe20",
      "status": "Failed",
      "startTime": "2026-10-07T20:41:47.534201418Z",
      "endTime": "2026-10-07T21:10:48.799809829Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9 to exist\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-tf4n9 to exist\nMachine o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cbh64xfpcppm-candidate-6rlkdh--managed/providers/Microsoft.Compute/virtualMachines/o2l4f8h5q8g0s9v-np-6rlkdh-k4ds7-zfbl8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kddk55" for cluster "cluster-candidate-5-1-kddk55"
Unexpected error:
    <*fmt.wrapError | 0x13bc8950a6a0>: 
    failed to create NodePool np-kddk55: failed waiting for nodepool="np-kddk55" for cluster "cluster-candidate-5-1-kddk55" in resourcegroup="rg-candidate-5-1-fsc4c2dfddtk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e435f098-79af-4365-b5b1-76c03affaa2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e435f098-79af-4365-b5b1-76c03affaa2b",
      "name": "e435f098-79af-4365-b5b1-76c03affaa2b",
      "status": "Failed",
      "startTime": "2026-10-07T20:43:26.129269305Z",
      "endTime": "2026-10-07T21:12:28.780512619Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8 to exist\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8 to exist\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kddk55" for cluster "cluster-candidate-5-1-kddk55"
Unexpected error:
    <*fmt.wrapError | 0x13bc8950a6a0>: 
    failed to create NodePool np-kddk55: failed waiting for nodepool="np-kddk55" for cluster "cluster-candidate-5-1-kddk55" in resourcegroup="rg-candidate-5-1-fsc4c2dfddtk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e435f098-79af-4365-b5b1-76c03affaa2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e435f098-79af-4365-b5b1-76c03affaa2b",
      "name": "e435f098-79af-4365-b5b1-76c03affaa2b",
      "status": "Failed",
      "startTime": "2026-10-07T20:43:26.129269305Z",
      "endTime": "2026-10-07T21:12:28.780512619Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8 to exist\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-2w9m8 to exist\nMachine q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-fsc4c2dfddtk-candidate-kddk55--managed/providers/Microsoft.Compute/virtualMachines/q5q7c9q9x4f1p2k-np-kddk55-tvdpc-h55rn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
19:33:35 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d0dde8c3780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cdl8czz92fjt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8",
      "name": "1b34011e-02fa-407e-bf8e-697c866145e8",
      "status": "Failed",
      "startTime": "2026-10-07T21:30:13.060183446Z",
      "endTime": "2026-10-07T21:59:13.201730463Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d0dde8c3780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cdl8czz92fjt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b34011e-02fa-407e-bf8e-697c866145e8",
      "name": "1b34011e-02fa-407e-bf8e-697c866145e8",
      "status": "Failed",
      "startTime": "2026-10-07T21:30:13.060183446Z",
      "endTime": "2026-10-07T21:59:13.201730463Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-qrr68 to exist\nMachine u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cdl8czz92fjt--managed/providers/Microsoft.Compute/virtualMachines/u7f6c6k9k0a0v7e-np-deletable-gh9lt-tbkcs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-97mck8" for cluster "cluster-candidate-4-22-97mck8"
Unexpected error:
    <*fmt.wrapError | 0x229053c35120>: 
    failed to create NodePool np-97mck8: failed waiting for nodepool="np-97mck8" for cluster "cluster-candidate-4-22-97mck8" in resourcegroup="rg-candidate-4-22-j7f8d9chl5df" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b172bf9-41e9-401d-953d-ee155521aa75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b172bf9-41e9-401d-953d-ee155521aa75",
      "name": "5b172bf9-41e9-401d-953d-ee155521aa75",
      "status": "Failed",
      "startTime": "2026-10-07T20:37:15.629028583Z",
      "endTime": "2026-10-07T21:06:18.880533991Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q to exist\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q to exist\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-97mck8" for cluster "cluster-candidate-4-22-97mck8"
Unexpected error:
    <*fmt.wrapError | 0x229053c35120>: 
    failed to create NodePool np-97mck8: failed waiting for nodepool="np-97mck8" for cluster "cluster-candidate-4-22-97mck8" in resourcegroup="rg-candidate-4-22-j7f8d9chl5df" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b172bf9-41e9-401d-953d-ee155521aa75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b172bf9-41e9-401d-953d-ee155521aa75",
      "name": "5b172bf9-41e9-401d-953d-ee155521aa75",
      "status": "Failed",
      "startTime": "2026-10-07T20:37:15.629028583Z",
      "endTime": "2026-10-07T21:06:18.880533991Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q to exist\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-b2j6q to exist\nMachine z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-j7f8d9chl5df-candidate-97mck8--managed/providers/Microsoft.Compute/virtualMachines/z8r3n7p4s1u6l5s-np-97mck8-h57hq-h6n6d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x32859edcc030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-bngnhgkwvt2t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ba4b9fd7-7f9c-4a19-89d1-ba7979c31bd3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ba4b9fd7-7f9c-4a19-89d1-ba7979c31bd3",
      "name": "ba4b9fd7-7f9c-4a19-89d1-ba7979c31bd3",
      "status": "Failed",
      "startTime": "2026-10-07T20:11:50.180436689Z",
      "endTime": "2026-10-07T20:40:50.55544909Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7x7rd-96k8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bngnhgkwvt2t--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7x7rd-96k8w to exist\nMachine agg-cluster-agg-np-a-7x7rd-bx6ks: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7x7rd-bx6ks: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7x7rd-96k8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bngnhgkwvt2t--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7x7rd-96k8w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x32859edcc030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-bngnhgkwvt2t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ba4b9fd7-7f9c-4a19-89d1-ba7979c31bd3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ba4b9fd7-7f9c-4a19-89d1-ba7979c31bd3",
      "name": "ba4b9fd7-7f9c-4a19-89d1-ba7979c31bd3",
      "status": "Failed",
      "startTime": "2026-10-07T20:11:50.180436689Z",
      "endTime": "2026-10-07T20:40:50.55544909Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7x7rd-96k8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bngnhgkwvt2t--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7x7rd-96k8w to exist\nMachine agg-cluster-agg-np-a-7x7rd-bx6ks: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7x7rd-bx6ks: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7x7rd-96k8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bngnhgkwvt2t--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7x7rd-96k8w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:33:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-42p26n" for cluster "cluster-candidate-5-0-42p26n"
Unexpected error:
    <*fmt.wrapError | 0x22a220e39e0>: 
    failed to create NodePool np-42p26n: failed waiting for nodepool="np-42p26n" for cluster "cluster-candidate-5-0-42p26n" in resourcegroup="rg-candidate-5-0-ggx4tjzkw6bv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4cfad558-f967-4b4f-bc63-df85c869948f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4cfad558-f967-4b4f-bc63-df85c869948f",
      "name": "4cfad558-f967-4b4f-bc63-df85c869948f",
      "status": "Failed",
      "startTime": "2026-10-07T20:41:47.770756531Z",
      "endTime": "2026-10-07T21:10:48.852402374Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc to exist\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc to exist\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-42p26n" for cluster "cluster-candidate-5-0-42p26n"
Unexpected error:
    <*fmt.wrapError | 0x22a220e39e0>: 
    failed to create NodePool np-42p26n: failed waiting for nodepool="np-42p26n" for cluster "cluster-candidate-5-0-42p26n" in resourcegroup="rg-candidate-5-0-ggx4tjzkw6bv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4cfad558-f967-4b4f-bc63-df85c869948f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4cfad558-f967-4b4f-bc63-df85c869948f",
      "name": "4cfad558-f967-4b4f-bc63-df85c869948f",
      "status": "Failed",
      "startTime": "2026-10-07T20:41:47.770756531Z",
      "endTime": "2026-10-07T21:10:48.852402374Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc to exist\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-594cc to exist\nMachine g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-ggx4tjzkw6bv-candidate-42p26n--managed/providers/Microsoft.Compute/virtualMachines/g1e2k3e1t3e6h3w-np-42p26n-gchjq-f9xb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 54m 39s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth e2e failed n/a 11
e2e (11)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
16:34:17 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x35ed21e504a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6xhlmlzjgvst" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/12991b47-b015-4cbd-9985-f7c3a34af217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/12991b47-b015-4cbd-9985-f7c3a34af217",
      "name": "12991b47-b015-4cbd-9985-f7c3a34af217",
      "status": "Failed",
      "startTime": "2026-10-07T18:35:02.297058773Z",
      "endTime": "2026-10-07T19:04:10.718534768Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t to exist\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t to exist\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x35ed21e504a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6xhlmlzjgvst" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/12991b47-b015-4cbd-9985-f7c3a34af217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/12991b47-b015-4cbd-9985-f7c3a34af217",
      "name": "12991b47-b015-4cbd-9985-f7c3a34af217",
      "status": "Failed",
      "startTime": "2026-10-07T18:35:02.297058773Z",
      "endTime": "2026-10-07T19:04:10.718534768Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t to exist\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-76h9t to exist\nMachine n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6xhlmlzjgvst--managed/providers/Microsoft.Compute/virtualMachines/n4a0f5c4d7s5h7j-np-deletable-sn575-blj2k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lscpsw" for cluster "cluster-candidate-5-1-lscpsw"
Unexpected error:
    <*fmt.wrapError | 0x2a793a7485a0>: 
    failed to create NodePool np-lscpsw: failed waiting for nodepool="np-lscpsw" for cluster "cluster-candidate-5-1-lscpsw" in resourcegroup="rg-candidate-5-1-xrcj42xlnjvs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5813dcb9-1643-4e9b-822d-a0503544e72d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5813dcb9-1643-4e9b-822d-a0503544e72d",
      "name": "5813dcb9-1643-4e9b-822d-a0503544e72d",
      "status": "Failed",
      "startTime": "2026-10-07T17:46:26.081906797Z",
      "endTime": "2026-10-07T18:15:34.783886022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg to exist\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg to exist\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lscpsw" for cluster "cluster-candidate-5-1-lscpsw"
Unexpected error:
    <*fmt.wrapError | 0x2a793a7485a0>: 
    failed to create NodePool np-lscpsw: failed waiting for nodepool="np-lscpsw" for cluster "cluster-candidate-5-1-lscpsw" in resourcegroup="rg-candidate-5-1-xrcj42xlnjvs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5813dcb9-1643-4e9b-822d-a0503544e72d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5813dcb9-1643-4e9b-822d-a0503544e72d",
      "name": "5813dcb9-1643-4e9b-822d-a0503544e72d",
      "status": "Failed",
      "startTime": "2026-10-07T17:46:26.081906797Z",
      "endTime": "2026-10-07T18:15:34.783886022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg to exist\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-b86dg to exist\nMachine c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-xrcj42xlnjvs-candidate-lscpsw--managed/providers/Microsoft.Compute/virtualMachines/c9q8s0r5e8m4v4l-np-lscpsw-xdz87-qxpvk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2kqzsv" for cluster "cluster-candidate-4-20-2kqzsv"
Unexpected error:
    <*fmt.wrapError | 0x1103da4b8e00>: 
    failed to create NodePool np-2kqzsv: failed waiting for nodepool="np-2kqzsv" for cluster "cluster-candidate-4-20-2kqzsv" in resourcegroup="rg-candidate-4-20-hcllc94k97c8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d9788fe4-7e50-4854-8d7b-82e7c83085c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d9788fe4-7e50-4854-8d7b-82e7c83085c6",
      "name": "d9788fe4-7e50-4854-8d7b-82e7c83085c6",
      "status": "Failed",
      "startTime": "2026-10-07T17:40:34.472182873Z",
      "endTime": "2026-10-07T18:09:34.723474878Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4 to exist\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4 to exist\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2kqzsv" for cluster "cluster-candidate-4-20-2kqzsv"
Unexpected error:
    <*fmt.wrapError | 0x1103da4b8e00>: 
    failed to create NodePool np-2kqzsv: failed waiting for nodepool="np-2kqzsv" for cluster "cluster-candidate-4-20-2kqzsv" in resourcegroup="rg-candidate-4-20-hcllc94k97c8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d9788fe4-7e50-4854-8d7b-82e7c83085c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d9788fe4-7e50-4854-8d7b-82e7c83085c6",
      "name": "d9788fe4-7e50-4854-8d7b-82e7c83085c6",
      "status": "Failed",
      "startTime": "2026-10-07T17:40:34.472182873Z",
      "endTime": "2026-10-07T18:09:34.723474878Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4 to exist\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-cr5n4 to exist\nMachine n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hcllc94k97c8-candidate-2kqzsv--managed/providers/Microsoft.Compute/virtualMachines/n7o5u6t2t9p0s8z-np-2kqzsv-9clt4-kfzjd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nb62wj" for cluster "cluster-candidate-4-23-nb62wj"
Unexpected error:
    <*fmt.wrapError | 0x3d66910451c0>: 
    failed to create NodePool np-nb62wj: failed waiting for nodepool="np-nb62wj" for cluster "cluster-candidate-4-23-nb62wj" in resourcegroup="rg-candidate-4-23-grg7vn74klw9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d174db36-7fc5-4f12-a2d6-87f012680f99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d174db36-7fc5-4f12-a2d6-87f012680f99",
      "name": "d174db36-7fc5-4f12-a2d6-87f012680f99",
      "status": "Failed",
      "startTime": "2026-10-07T17:44:23.305815076Z",
      "endTime": "2026-10-07T18:13:24.731351337Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r to exist\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r to exist\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nb62wj" for cluster "cluster-candidate-4-23-nb62wj"
Unexpected error:
    <*fmt.wrapError | 0x3d66910451c0>: 
    failed to create NodePool np-nb62wj: failed waiting for nodepool="np-nb62wj" for cluster "cluster-candidate-4-23-nb62wj" in resourcegroup="rg-candidate-4-23-grg7vn74klw9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d174db36-7fc5-4f12-a2d6-87f012680f99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d174db36-7fc5-4f12-a2d6-87f012680f99",
      "name": "d174db36-7fc5-4f12-a2d6-87f012680f99",
      "status": "Failed",
      "startTime": "2026-10-07T17:44:23.305815076Z",
      "endTime": "2026-10-07T18:13:24.731351337Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r to exist\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-7m58r to exist\nMachine w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-grg7vn74klw9-candidate-nb62wj--managed/providers/Microsoft.Compute/virtualMachines/w6t0v4x4s5n3m8e-np-nb62wj-wp9jt-8mkbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
16:34:17 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x13d15dfbe0a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-pzwq2wsxtbjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eff75568-ebfe-4e11-bf54-098440849979
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eff75568-ebfe-4e11-bf54-098440849979",
      "name": "eff75568-ebfe-4e11-bf54-098440849979",
      "status": "Failed",
      "startTime": "2026-10-07T17:15:36.806023423Z",
      "endTime": "2026-10-07T17:44:44.250099219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-m6k6m-h4k42: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-pzwq2wsxtbjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m6k6m-h4k42 to exist\nMachine agg-cluster-agg-np-a-m6k6m-wqcdr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-m6k6m-wqcdr: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-m6k6m-h4k42: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-pzwq2wsxtbjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m6k6m-h4k42 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x13d15dfbe0a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-pzwq2wsxtbjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eff75568-ebfe-4e11-bf54-098440849979
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eff75568-ebfe-4e11-bf54-098440849979",
      "name": "eff75568-ebfe-4e11-bf54-098440849979",
      "status": "Failed",
      "startTime": "2026-10-07T17:15:36.806023423Z",
      "endTime": "2026-10-07T17:44:44.250099219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-m6k6m-h4k42: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-pzwq2wsxtbjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m6k6m-h4k42 to exist\nMachine agg-cluster-agg-np-a-m6k6m-wqcdr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-m6k6m-wqcdr: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-m6k6m-h4k42: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-pzwq2wsxtbjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-m6k6m-h4k42 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x206ba567a560>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5jrlb2b5r7kp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b896af0-0d09-4a15-8405-35074545d1b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b896af0-0d09-4a15-8405-35074545d1b6",
      "name": "1b896af0-0d09-4a15-8405-35074545d1b6",
      "status": "Failed",
      "startTime": "2026-10-07T17:13:20.764000805Z",
      "endTime": "2026-10-07T17:42:24.218659918Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-f74v6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-f74v6 to exist\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-tbspd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-tbspd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-f74v6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-f74v6 to exist\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-tbspd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-tbspd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x206ba567a560>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5jrlb2b5r7kp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b896af0-0d09-4a15-8405-35074545d1b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1b896af0-0d09-4a15-8405-35074545d1b6",
      "name": "1b896af0-0d09-4a15-8405-35074545d1b6",
      "status": "Failed",
      "startTime": "2026-10-07T17:13:20.764000805Z",
      "endTime": "2026-10-07T17:42:24.218659918Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-f74v6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-f74v6 to exist\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-tbspd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-tbspd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-f74v6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-f74v6 to exist\nMachine j4c7y2m5q3o0j6f-np-1-szwxx-tbspd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5jrlb2b5r7kp--managed/providers/Microsoft.Compute/virtualMachines/j4c7y2m5q3o0j6f-np-1-szwxx-tbspd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lcwd68" for cluster "cluster-candidate-4-21-lcwd68"
Unexpected error:
    <*fmt.wrapError | 0x3c328d6f3460>: 
    failed to create NodePool np-lcwd68: failed waiting for nodepool="np-lcwd68" for cluster "cluster-candidate-4-21-lcwd68" in resourcegroup="rg-candidate-4-21-rrhmm78jhkht" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afb758ed-7109-4c7f-9f48-4c3c8f3bcd88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afb758ed-7109-4c7f-9f48-4c3c8f3bcd88",
      "name": "afb758ed-7109-4c7f-9f48-4c3c8f3bcd88",
      "status": "Failed",
      "startTime": "2026-10-07T17:43:13.338439783Z",
      "endTime": "2026-10-07T18:12:14.670254845Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv to exist\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv to exist\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lcwd68" for cluster "cluster-candidate-4-21-lcwd68"
Unexpected error:
    <*fmt.wrapError | 0x3c328d6f3460>: 
    failed to create NodePool np-lcwd68: failed waiting for nodepool="np-lcwd68" for cluster "cluster-candidate-4-21-lcwd68" in resourcegroup="rg-candidate-4-21-rrhmm78jhkht" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afb758ed-7109-4c7f-9f48-4c3c8f3bcd88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afb758ed-7109-4c7f-9f48-4c3c8f3bcd88",
      "name": "afb758ed-7109-4c7f-9f48-4c3c8f3bcd88",
      "status": "Failed",
      "startTime": "2026-10-07T17:43:13.338439783Z",
      "endTime": "2026-10-07T18:12:14.670254845Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv to exist\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-gxzgv to exist\nMachine b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rrhmm78jhkht-candidate-lcwd68--managed/providers/Microsoft.Compute/virtualMachines/b5c7e1e4t0q5v9a-np-lcwd68-4bprc-zwhlg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-p7gdrb" for cluster "cluster-candidate-5-0-p7gdrb"
Unexpected error:
    <*fmt.wrapError | 0x2fb1d5f71ba0>: 
    failed to create NodePool np-p7gdrb: failed waiting for nodepool="np-p7gdrb" for cluster "cluster-candidate-5-0-p7gdrb" in resourcegroup="rg-candidate-5-0-pc4wqgt64lq4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4c46141b-f29e-4146-896e-882152b7eceb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4c46141b-f29e-4146-896e-882152b7eceb",
      "name": "4c46141b-f29e-4146-896e-882152b7eceb",
      "status": "Failed",
      "startTime": "2026-10-07T17:45:14.537982049Z",
      "endTime": "2026-10-07T18:14:14.681314604Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb to exist\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb to exist\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-p7gdrb" for cluster "cluster-candidate-5-0-p7gdrb"
Unexpected error:
    <*fmt.wrapError | 0x2fb1d5f71ba0>: 
    failed to create NodePool np-p7gdrb: failed waiting for nodepool="np-p7gdrb" for cluster "cluster-candidate-5-0-p7gdrb" in resourcegroup="rg-candidate-5-0-pc4wqgt64lq4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4c46141b-f29e-4146-896e-882152b7eceb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4c46141b-f29e-4146-896e-882152b7eceb",
      "name": "4c46141b-f29e-4146-896e-882152b7eceb",
      "status": "Failed",
      "startTime": "2026-10-07T17:45:14.537982049Z",
      "endTime": "2026-10-07T18:14:14.681314604Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb to exist\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-59vxb to exist\nMachine p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pc4wqgt64lq4-candidate-p7gdrb--managed/providers/Microsoft.Compute/virtualMachines/p2s0l6s4q6p5x7g-np-p7gdrb-pqx5j-g9q8s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x195b047b05e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-hkmgcjdzqcqq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bdff2772-3fa6-4abc-9d11-2c48138ff9bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bdff2772-3fa6-4abc-9d11-2c48138ff9bc",
      "name": "bdff2772-3fa6-4abc-9d11-2c48138ff9bc",
      "status": "Failed",
      "startTime": "2026-10-07T17:14:56.224189159Z",
      "endTime": "2026-10-07T17:44:04.263637459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-jttqc-rd98n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-hkmgcjdzqcqq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-jttqc-rd98n to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-jttqc-rd98n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-hkmgcjdzqcqq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-jttqc-rd98n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x195b047b05e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-hkmgcjdzqcqq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bdff2772-3fa6-4abc-9d11-2c48138ff9bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/bdff2772-3fa6-4abc-9d11-2c48138ff9bc",
      "name": "bdff2772-3fa6-4abc-9d11-2c48138ff9bc",
      "status": "Failed",
      "startTime": "2026-10-07T17:14:56.224189159Z",
      "endTime": "2026-10-07T17:44:04.263637459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-jttqc-rd98n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-hkmgcjdzqcqq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-jttqc-rd98n to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-jttqc-rd98n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-hkmgcjdzqcqq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-jttqc-rd98n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x21e4baf3c498>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-gpqfttn5vrxt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0bfed0f8-a294-4f34-9bc1-112faf5a4865
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0bfed0f8-a294-4f34-9bc1-112faf5a4865",
      "name": "0bfed0f8-a294-4f34-9bc1-112faf5a4865",
      "status": "Failed",
      "startTime": "2026-10-07T17:15:04.407912197Z",
      "endTime": "2026-10-07T17:44:04.236306088Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-wcplc-b8dk7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-b8dk7 to exist\nMachine cilium-cluster-cilium-np-wcplc-spjl7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-spjl7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-wcplc-b8dk7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-b8dk7 to exist\nMachine cilium-cluster-cilium-np-wcplc-spjl7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-spjl7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x21e4baf3c498>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-gpqfttn5vrxt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0bfed0f8-a294-4f34-9bc1-112faf5a4865
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0bfed0f8-a294-4f34-9bc1-112faf5a4865",
      "name": "0bfed0f8-a294-4f34-9bc1-112faf5a4865",
      "status": "Failed",
      "startTime": "2026-10-07T17:15:04.407912197Z",
      "endTime": "2026-10-07T17:44:04.236306088Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-wcplc-b8dk7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-b8dk7 to exist\nMachine cilium-cluster-cilium-np-wcplc-spjl7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-spjl7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-wcplc-b8dk7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-b8dk7 to exist\nMachine cilium-cluster-cilium-np-wcplc-spjl7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-gpqfttn5vrxt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-wcplc-spjl7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:34:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cq8x8c" for cluster "cluster-candidate-4-22-cq8x8c"
Unexpected error:
    <*fmt.wrapError | 0x1b2b51738800>: 
    failed to create NodePool np-cq8x8c: failed waiting for nodepool="np-cq8x8c" for cluster "cluster-candidate-4-22-cq8x8c" in resourcegroup="rg-candidate-4-22-jv6xxmcszfmz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dc58718-cb83-4ae7-9c5f-3112077aac0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dc58718-cb83-4ae7-9c5f-3112077aac0f",
      "name": "5dc58718-cb83-4ae7-9c5f-3112077aac0f",
      "status": "Failed",
      "startTime": "2026-10-07T17:43:42.061217381Z",
      "endTime": "2026-10-07T18:12:44.656595031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892 to exist\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892 to exist\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cq8x8c" for cluster "cluster-candidate-4-22-cq8x8c"
Unexpected error:
    <*fmt.wrapError | 0x1b2b51738800>: 
    failed to create NodePool np-cq8x8c: failed waiting for nodepool="np-cq8x8c" for cluster "cluster-candidate-4-22-cq8x8c" in resourcegroup="rg-candidate-4-22-jv6xxmcszfmz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dc58718-cb83-4ae7-9c5f-3112077aac0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5dc58718-cb83-4ae7-9c5f-3112077aac0f",
      "name": "5dc58718-cb83-4ae7-9c5f-3112077aac0f",
      "status": "Failed",
      "startTime": "2026-10-07T17:43:42.061217381Z",
      "endTime": "2026-10-07T18:12:44.656595031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892 to exist\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-cz892 to exist\nMachine q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-jv6xxmcszfmz-candidate-cq8x8c--managed/providers/Microsoft.Compute/virtualMachines/q1a5u8i6s7q3g7y-np-cq8x8c-hpdc9-tscrv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
3h 14m 16s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 eastus2euap unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
3h 19m 56s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a e2e failed n/a 60
e2e (60)
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-6g2vplsk2mk7/cluster-candidate-5-0-h6qmxk should provision
Unexpected error:
    <*fmt.wrapError | 0x927ddf70460>: 
    failed to create HCP cluster cluster-candidate-5-0-h6qmxk: failed waiting for cluster="cluster-candidate-5-0-h6qmxk" in resourcegroup="rg-candidate-5-0-6g2vplsk2mk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f5559ea-24ce-464c-ba42-f4702aefacd5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f5559ea-24ce-464c-ba42-f4702aefacd5",
      "name": "9f5559ea-24ce-464c-ba42-f4702aefacd5",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:14.87447448Z",
      "endTime": "2026-10-07T11:13:14.934598428Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-6g2vplsk2mk7/cluster-candidate-5-0-h6qmxk should provision
Unexpected error:
    <*fmt.wrapError | 0x927ddf70460>: 
    failed to create HCP cluster cluster-candidate-5-0-h6qmxk: failed waiting for cluster="cluster-candidate-5-0-h6qmxk" in resourcegroup="rg-candidate-5-0-6g2vplsk2mk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f5559ea-24ce-464c-ba42-f4702aefacd5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f5559ea-24ce-464c-ba42-f4702aefacd5",
      "name": "9f5559ea-24ce-464c-ba42-f4702aefacd5",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:14.87447448Z",
      "endTime": "2026-10-07T11:13:14.934598428Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x341fe2b86760>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-ptq5zsv79s42" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/53b78cd8-ac9a-4444-a5d8-4c19b93bf968
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/53b78cd8-ac9a-4444-a5d8-4c19b93bf968",
      "name": "53b78cd8-ac9a-4444-a5d8-4c19b93bf968",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:55.810865072Z",
      "endTime": "2026-10-07T10:18:04.487764754Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x341fe2b86760>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-ptq5zsv79s42" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/53b78cd8-ac9a-4444-a5d8-4c19b93bf968
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/53b78cd8-ac9a-4444-a5d8-4c19b93bf968",
      "name": "53b78cd8-ac9a-4444-a5d8-4c19b93bf968",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:55.810865072Z",
      "endTime": "2026-10-07T10:18:04.487764754Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x186c1f59fe00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-xkrgtz49hrh4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xkrgtz49hrh4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x186c1f59fe00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-xkrgtz49hrh4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xkrgtz49hrh4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x20bebe634820>: 
    failed to create HCP cluster ea-listrlzp2p: failed waiting for cluster="ea-listrlzp2p" in resourcegroup="ea-list-k4w9b29d9847" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07411f45-babd-4a30-b6ee-3bba10d43fa1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07411f45-babd-4a30-b6ee-3bba10d43fa1",
      "name": "07411f45-babd-4a30-b6ee-3bba10d43fa1",
      "status": "Failed",
      "startTime": "2026-10-07T10:47:32.468226037Z",
      "endTime": "2026-10-07T11:16:34.835170504Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x20bebe634820>: 
    failed to create HCP cluster ea-listrlzp2p: failed waiting for cluster="ea-listrlzp2p" in resourcegroup="ea-list-k4w9b29d9847" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07411f45-babd-4a30-b6ee-3bba10d43fa1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07411f45-babd-4a30-b6ee-3bba10d43fa1",
      "name": "07411f45-babd-4a30-b6ee-3bba10d43fa1",
      "status": "Failed",
      "startTime": "2026-10-07T10:47:32.468226037Z",
      "endTime": "2026-10-07T11:16:34.835170504Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x15d5627c8480>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-jq7pzjw2cp54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6fe1a352-ffa7-4b41-9c47-d5903779eadf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6fe1a352-ffa7-4b41-9c47-d5903779eadf",
      "name": "6fe1a352-ffa7-4b41-9c47-d5903779eadf",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:08.895469535Z",
      "endTime": "2026-10-07T10:21:14.61315509Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x15d5627c8480>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-jq7pzjw2cp54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6fe1a352-ffa7-4b41-9c47-d5903779eadf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6fe1a352-ffa7-4b41-9c47-d5903779eadf",
      "name": "6fe1a352-ffa7-4b41-9c47-d5903779eadf",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:08.895469535Z",
      "endTime": "2026-10-07T10:21:14.61315509Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xde11fb947e0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0c5a96-73a7-46a1-ae56-141bff5a9cb0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0c5a96-73a7-46a1-ae56-141bff5a9cb0",
      "name": "3c0c5a96-73a7-46a1-ae56-141bff5a9cb0",
      "status": "Failed",
      "startTime": "2026-10-07T09:49:41.529698029Z",
      "endTime": "2026-10-07T10:18:44.600218991Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xde11fb947e0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0c5a96-73a7-46a1-ae56-141bff5a9cb0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0c5a96-73a7-46a1-ae56-141bff5a9cb0",
      "name": "3c0c5a96-73a7-46a1-ae56-141bff5a9cb0",
      "status": "Failed",
      "startTime": "2026-10-07T09:49:41.529698029Z",
      "endTime": "2026-10-07T10:18:44.600218991Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3437203728c0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mgbhjw52mcvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e9424b8-7f1c-40c2-82b4-31517d779d0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e9424b8-7f1c-40c2-82b4-31517d779d0b",
      "name": "6e9424b8-7f1c-40c2-82b4-31517d779d0b",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:54.433534766Z",
      "endTime": "2026-10-07T10:17:54.588329133Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3437203728c0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mgbhjw52mcvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e9424b8-7f1c-40c2-82b4-31517d779d0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e9424b8-7f1c-40c2-82b4-31517d779d0b",
      "name": "6e9424b8-7f1c-40c2-82b4-31517d779d0b",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:54.433534766Z",
      "endTime": "2026-10-07T10:17:54.588329133Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x14682ff371e0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5tccgln57r4g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cae4d34-c9f9-4fce-84cc-e6e26de3b174
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cae4d34-c9f9-4fce-84cc-e6e26de3b174",
      "name": "5cae4d34-c9f9-4fce-84cc-e6e26de3b174",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:14.268574552Z",
      "endTime": "2026-10-07T12:08:14.728480249Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x14682ff371e0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5tccgln57r4g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cae4d34-c9f9-4fce-84cc-e6e26de3b174
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5cae4d34-c9f9-4fce-84cc-e6e26de3b174",
      "name": "5cae4d34-c9f9-4fce-84cc-e6e26de3b174",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:14.268574552Z",
      "endTime": "2026-10-07T12:08:14.728480249Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x2e0fc50a4480>: 
    failed to create HCP cluster np-downgrade-s292rk: failed waiting for cluster="np-downgrade-s292rk" in resourcegroup="rg-np-downgrade-s292rk-zk4jmh695nzj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c877c7b1-118a-45af-95c5-bf8e831b2f1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c877c7b1-118a-45af-95c5-bf8e831b2f1d",
      "name": "c877c7b1-118a-45af-95c5-bf8e831b2f1d",
      "status": "Failed",
      "startTime": "2026-10-07T11:40:59.843518116Z",
      "endTime": "2026-10-07T12:10:04.833949367Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x2e0fc50a4480>: 
    failed to create HCP cluster np-downgrade-s292rk: failed waiting for cluster="np-downgrade-s292rk" in resourcegroup="rg-np-downgrade-s292rk-zk4jmh695nzj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c877c7b1-118a-45af-95c5-bf8e831b2f1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c877c7b1-118a-45af-95c5-bf8e831b2f1d",
      "name": "c877c7b1-118a-45af-95c5-bf8e831b2f1d",
      "status": "Failed",
      "startTime": "2026-10-07T11:40:59.843518116Z",
      "endTime": "2026-10-07T12:10:04.833949367Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2851e862a620>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ws854cfjbfc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "name": "5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:27.70826866Z",
      "endTime": "2026-10-07T12:10:34.84968826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2851e862a620>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ws854cfjbfc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "name": "5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:27.70826866Z",
      "endTime": "2026-10-07T12:10:34.84968826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x5e92b4ffb60>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-jsrm64gc6pjs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-jsrm64gc6pjs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x5e92b4ffb60>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-jsrm64gc6pjs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-jsrm64gc6pjs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x37dd151f8520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-6cgsv65nx9n7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/34b0c2af-8aaa-4f79-9413-e75d8e7d93a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/34b0c2af-8aaa-4f79-9413-e75d8e7d93a2",
      "name": "34b0c2af-8aaa-4f79-9413-e75d8e7d93a2",
      "status": "Failed",
      "startTime": "2026-10-07T09:50:14.090475934Z",
      "endTime": "2026-10-07T10:19:14.520219506Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x37dd151f8520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-6cgsv65nx9n7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/34b0c2af-8aaa-4f79-9413-e75d8e7d93a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/34b0c2af-8aaa-4f79-9413-e75d8e7d93a2",
      "name": "34b0c2af-8aaa-4f79-9413-e75d8e7d93a2",
      "status": "Failed",
      "startTime": "2026-10-07T09:50:14.090475934Z",
      "endTime": "2026-10-07T10:19:14.520219506Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xb405b54e780>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-xwt6pngn9s2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39973bec-a044-461c-a50c-080b9b4ce44e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39973bec-a044-461c-a50c-080b9b4ce44e",
      "name": "39973bec-a044-461c-a50c-080b9b4ce44e",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:24.877409448Z",
      "endTime": "2026-10-07T12:08:24.807664199Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xb405b54e780>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-xwt6pngn9s2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39973bec-a044-461c-a50c-080b9b4ce44e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39973bec-a044-461c-a50c-080b9b4ce44e",
      "name": "39973bec-a044-461c-a50c-080b9b4ce44e",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:24.877409448Z",
      "endTime": "2026-10-07T12:08:24.807664199Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x231265410c40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-m6pl2gptcdfp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "name": "3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:27.398564047Z",
      "endTime": "2026-10-07T12:08:34.791319585Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x231265410c40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-m6pl2gptcdfp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "name": "3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:27.398564047Z",
      "endTime": "2026-10-07T12:08:34.791319585Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkdz2nxrsjhh/cluster-candidate-4-21-d4ssjn should provision
Unexpected error:
    <*fmt.wrapError | 0x38f8a2b10240>: 
    failed to create HCP cluster cluster-candidate-4-21-d4ssjn: failed waiting for cluster="cluster-candidate-4-21-d4ssjn" in resourcegroup="rg-candidate-4-21-hkdz2nxrsjhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3579b35f-53ea-4026-90cb-38b7f7202820
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3579b35f-53ea-4026-90cb-38b7f7202820",
      "name": "3579b35f-53ea-4026-90cb-38b7f7202820",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.416842426Z",
      "endTime": "2026-10-07T11:13:04.946994538Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkdz2nxrsjhh/cluster-candidate-4-21-d4ssjn should provision
Unexpected error:
    <*fmt.wrapError | 0x38f8a2b10240>: 
    failed to create HCP cluster cluster-candidate-4-21-d4ssjn: failed waiting for cluster="cluster-candidate-4-21-d4ssjn" in resourcegroup="rg-candidate-4-21-hkdz2nxrsjhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3579b35f-53ea-4026-90cb-38b7f7202820
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3579b35f-53ea-4026-90cb-38b7f7202820",
      "name": "3579b35f-53ea-4026-90cb-38b7f7202820",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.416842426Z",
      "endTime": "2026-10-07T11:13:04.946994538Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x335c22036160>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-7nldjt8fwf95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff0ed7f2-6772-4f0b-98e5-ec67a1f2a5e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff0ed7f2-6772-4f0b-98e5-ec67a1f2a5e4",
      "name": "ff0ed7f2-6772-4f0b-98e5-ec67a1f2a5e4",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:28.998485099Z",
      "endTime": "2026-10-07T10:21:34.495305491Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x335c22036160>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-7nldjt8fwf95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff0ed7f2-6772-4f0b-98e5-ec67a1f2a5e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff0ed7f2-6772-4f0b-98e5-ec67a1f2a5e4",
      "name": "ff0ed7f2-6772-4f0b-98e5-ec67a1f2a5e4",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:28.998485099Z",
      "endTime": "2026-10-07T10:21:34.495305491Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0x30510e393840>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-6n2m8kjftrjk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f40a774-0a6b-406f-b3e0-c881c5b48c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f40a774-0a6b-406f-b3e0-c881c5b48c28",
      "name": "1f40a774-0a6b-406f-b3e0-c881c5b48c28",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:52.605546648Z",
      "endTime": "2026-10-07T10:17:54.505358431Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0x30510e393840>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-6n2m8kjftrjk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f40a774-0a6b-406f-b3e0-c881c5b48c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f40a774-0a6b-406f-b3e0-c881c5b48c28",
      "name": "1f40a774-0a6b-406f-b3e0-c881c5b48c28",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:52.605546648Z",
      "endTime": "2026-10-07T10:17:54.505358431Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-hjp4bn" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2a46bd3ea760>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-hjp4bn: failed waiting for cluster="cp-ystream-upgrade-4-22-hjp4bn" in resourcegroup="rg-cp-ystream-upgrade-4-22-69pb6s9zfqhj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5b982c67-2a0b-4195-a11b-235682400cd2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5b982c67-2a0b-4195-a11b-235682400cd2",
      "name": "5b982c67-2a0b-4195-a11b-235682400cd2",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:31.219755391Z",
      "endTime": "2026-10-07T11:15:34.961415579Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-hjp4bn" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2a46bd3ea760>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-hjp4bn: failed waiting for cluster="cp-ystream-upgrade-4-22-hjp4bn" in resourcegroup="rg-cp-ystream-upgrade-4-22-69pb6s9zfqhj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5b982c67-2a0b-4195-a11b-235682400cd2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5b982c67-2a0b-4195-a11b-235682400cd2",
      "name": "5b982c67-2a0b-4195-a11b-235682400cd2",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:31.219755391Z",
      "endTime": "2026-10-07T11:15:34.961415579Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-tqxzz2 at version 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x331f2f68e6a0>: 
    failed to create HCP cluster np-skip-minor-tqxzz2: failed waiting for cluster="np-skip-minor-tqxzz2" in resourcegroup="rg-np-skip-minor-tqxzz2-4phhgkzvl662" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3bf832de-8a79-4bb8-8b93-10f2de10820f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3bf832de-8a79-4bb8-8b93-10f2de10820f",
      "name": "3bf832de-8a79-4bb8-8b93-10f2de10820f",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:52.09303905Z",
      "endTime": "2026-10-07T12:08:54.828044154Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-tqxzz2 at version 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x331f2f68e6a0>: 
    failed to create HCP cluster np-skip-minor-tqxzz2: failed waiting for cluster="np-skip-minor-tqxzz2" in resourcegroup="rg-np-skip-minor-tqxzz2-4phhgkzvl662" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3bf832de-8a79-4bb8-8b93-10f2de10820f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3bf832de-8a79-4bb8-8b93-10f2de10820f",
      "name": "3bf832de-8a79-4bb8-8b93-10f2de10820f",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:52.09303905Z",
      "endTime": "2026-10-07T12:08:54.828044154Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-tcv2dg" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3a4975a7c5e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-tcv2dg: failed waiting for cluster="cp-ystream-upgrade-4-21-tcv2dg" in resourcegroup="rg-cp-ystream-upgrade-4-21-l28vh755d899" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3c3d8ab-c656-42f4-9688-9bfce51a0648
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3c3d8ab-c656-42f4-9688-9bfce51a0648",
      "name": "c3c3d8ab-c656-42f4-9688-9bfce51a0648",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:26.13173324Z",
      "endTime": "2026-10-07T11:15:34.96005862Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-tcv2dg" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3a4975a7c5e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-tcv2dg: failed waiting for cluster="cp-ystream-upgrade-4-21-tcv2dg" in resourcegroup="rg-cp-ystream-upgrade-4-21-l28vh755d899" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3c3d8ab-c656-42f4-9688-9bfce51a0648
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3c3d8ab-c656-42f4-9688-9bfce51a0648",
      "name": "c3c3d8ab-c656-42f4-9688-9bfce51a0648",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:26.13173324Z",
      "endTime": "2026-10-07T11:15:34.96005862Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x21855c06aa20>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-2bhn4hmftg4v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-2bhn4hmftg4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x21855c06aa20>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-2bhn4hmftg4v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-2bhn4hmftg4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-p6ctw2 with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x26e4dc71a4c0>: 
    failed to create HCP cluster np-version-upgrade-cluster-p6ctw2: failed waiting for cluster="np-version-upgrade-cluster-p6ctw2" in resourcegroup="rg-np-version-upgrade-p6ctw2-gxs4lhjp4gfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f6ade4-81d1-451d-b59d-15dcae59714f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f6ade4-81d1-451d-b59d-15dcae59714f",
      "name": "13f6ade4-81d1-451d-b59d-15dcae59714f",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:22.119909633Z",
      "endTime": "2026-10-07T12:08:24.732534106Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-p6ctw2 with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x26e4dc71a4c0>: 
    failed to create HCP cluster np-version-upgrade-cluster-p6ctw2: failed waiting for cluster="np-version-upgrade-cluster-p6ctw2" in resourcegroup="rg-np-version-upgrade-p6ctw2-gxs4lhjp4gfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f6ade4-81d1-451d-b59d-15dcae59714f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f6ade4-81d1-451d-b59d-15dcae59714f",
      "name": "13f6ade4-81d1-451d-b59d-15dcae59714f",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:22.119909633Z",
      "endTime": "2026-10-07T12:08:24.732534106Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z9xhpt9dxztr/cluster-candidate-4-22-8frl8d should provision
Unexpected error:
    <*fmt.wrapError | 0xc3c4040eec0>: 
    failed to create HCP cluster cluster-candidate-4-22-8frl8d: failed waiting for cluster="cluster-candidate-4-22-8frl8d" in resourcegroup="rg-candidate-4-22-z9xhpt9dxztr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "name": "6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.12350471Z",
      "endTime": "2026-10-07T11:13:04.942313802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z9xhpt9dxztr/cluster-candidate-4-22-8frl8d should provision
Unexpected error:
    <*fmt.wrapError | 0xc3c4040eec0>: 
    failed to create HCP cluster cluster-candidate-4-22-8frl8d: failed waiting for cluster="cluster-candidate-4-22-8frl8d" in resourcegroup="rg-candidate-4-22-z9xhpt9dxztr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "name": "6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.12350471Z",
      "endTime": "2026-10-07T11:13:04.942313802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x36474092e220>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-tsgkj9sbfmk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5c675833-7374-4d56-8296-8d479088785e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5c675833-7374-4d56-8296-8d479088785e",
      "name": "5c675833-7374-4d56-8296-8d479088785e",
      "status": "Failed",
      "startTime": "2026-10-07T09:51:43.574661454Z",
      "endTime": "2026-10-07T10:20:44.509181086Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x36474092e220>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-tsgkj9sbfmk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5c675833-7374-4d56-8296-8d479088785e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5c675833-7374-4d56-8296-8d479088785e",
      "name": "5c675833-7374-4d56-8296-8d479088785e",
      "status": "Failed",
      "startTime": "2026-10-07T09:51:43.574661454Z",
      "endTime": "2026-10-07T10:20:44.509181086Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-v64v5n" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3eb6842062a0>: 
    failed to create HCP cluster cluster-zstream-4-23-v64v5n: failed waiting for cluster="cluster-zstream-4-23-v64v5n" in resourcegroup="rg-zstream-upgrade-4-23-lxr77fqqlcc4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcd5b8-e1cc-4dde-aa1d-4fd166ab6ab6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcd5b8-e1cc-4dde-aa1d-4fd166ab6ab6",
      "name": "b0fcd5b8-e1cc-4dde-aa1d-4fd166ab6ab6",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:37.55941311Z",
      "endTime": "2026-10-07T11:13:44.839201363Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-v64v5n" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3eb6842062a0>: 
    failed to create HCP cluster cluster-zstream-4-23-v64v5n: failed waiting for cluster="cluster-zstream-4-23-v64v5n" in resourcegroup="rg-zstream-upgrade-4-23-lxr77fqqlcc4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcd5b8-e1cc-4dde-aa1d-4fd166ab6ab6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcd5b8-e1cc-4dde-aa1d-4fd166ab6ab6",
      "name": "b0fcd5b8-e1cc-4dde-aa1d-4fd166ab6ab6",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:37.55941311Z",
      "endTime": "2026-10-07T11:13:44.839201363Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-tvjfv8 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x566e5e2e580>: 
    failed to create HCP cluster np-version-upgrade-cluster-tvjfv8: failed waiting for cluster="np-version-upgrade-cluster-tvjfv8" in resourcegroup="rg-np-version-upgrade-tvjfv8-hct95xpq2lzg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f9a95eb-a350-40c6-af2a-e703473e0215
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f9a95eb-a350-40c6-af2a-e703473e0215",
      "name": "1f9a95eb-a350-40c6-af2a-e703473e0215",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:41.966281744Z",
      "endTime": "2026-10-07T12:08:44.804380288Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-tvjfv8 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x566e5e2e580>: 
    failed to create HCP cluster np-version-upgrade-cluster-tvjfv8: failed waiting for cluster="np-version-upgrade-cluster-tvjfv8" in resourcegroup="rg-np-version-upgrade-tvjfv8-hct95xpq2lzg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f9a95eb-a350-40c6-af2a-e703473e0215
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f9a95eb-a350-40c6-af2a-e703473e0215",
      "name": "1f9a95eb-a350-40c6-af2a-e703473e0215",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:41.966281744Z",
      "endTime": "2026-10-07T12:08:44.804380288Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-xfwltf
Unexpected error:
    <*fmt.wrapError | 0x22a102b44380>: 
    failed to create HCP cluster np-noedge-xfwltf: failed waiting for cluster="np-noedge-xfwltf" in resourcegroup="rg-np-noedge-xfwltf-wckscwtz2vn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75fdac03-68fe-4275-bc47-120c356d3ac3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75fdac03-68fe-4275-bc47-120c356d3ac3",
      "name": "75fdac03-68fe-4275-bc47-120c356d3ac3",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:51.394799188Z",
      "endTime": "2026-10-07T12:08:54.815579704Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-xfwltf
Unexpected error:
    <*fmt.wrapError | 0x22a102b44380>: 
    failed to create HCP cluster np-noedge-xfwltf: failed waiting for cluster="np-noedge-xfwltf" in resourcegroup="rg-np-noedge-xfwltf-wckscwtz2vn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75fdac03-68fe-4275-bc47-120c356d3ac3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75fdac03-68fe-4275-bc47-120c356d3ac3",
      "name": "75fdac03-68fe-4275-bc47-120c356d3ac3",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:51.394799188Z",
      "endTime": "2026-10-07T12:08:54.815579704Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0x1b5f89a004e0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-nkbzqbx5622d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5a305a1c-863d-4c3c-8dfe-d7cc2d61fb87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5a305a1c-863d-4c3c-8dfe-d7cc2d61fb87",
      "name": "5a305a1c-863d-4c3c-8dfe-d7cc2d61fb87",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:09.506899414Z",
      "endTime": "2026-10-07T10:21:14.622216782Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0x1b5f89a004e0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-nkbzqbx5622d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5a305a1c-863d-4c3c-8dfe-d7cc2d61fb87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5a305a1c-863d-4c3c-8dfe-d7cc2d61fb87",
      "name": "5a305a1c-863d-4c3c-8dfe-d7cc2d61fb87",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:09.506899414Z",
      "endTime": "2026-10-07T10:21:14.622216782Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x39ef57d831e0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-d2rzc85bvm5t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6397f521-d69c-40cd-8708-e9c85c21063d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6397f521-d69c-40cd-8708-e9c85c21063d",
      "name": "6397f521-d69c-40cd-8708-e9c85c21063d",
      "status": "Failed",
      "startTime": "2026-10-07T10:48:24.801806415Z",
      "endTime": "2026-10-07T11:17:24.840938454Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x39ef57d831e0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-d2rzc85bvm5t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6397f521-d69c-40cd-8708-e9c85c21063d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6397f521-d69c-40cd-8708-e9c85c21063d",
      "name": "6397f521-d69c-40cd-8708-e9c85c21063d",
      "status": "Failed",
      "startTime": "2026-10-07T10:48:24.801806415Z",
      "endTime": "2026-10-07T11:17:24.840938454Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x664b54ecae0>: 
    failed to create HCP cluster np-dg-minor-v6wk6c: failed waiting for cluster="np-dg-minor-v6wk6c" in resourcegroup="rg-np-dg-minor-v6wk6c-j7kpgqfkx4ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62536149-77a5-4b5d-a914-13bf7d06200d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62536149-77a5-4b5d-a914-13bf7d06200d",
      "name": "62536149-77a5-4b5d-a914-13bf7d06200d",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:24.096113734Z",
      "endTime": "2026-10-07T12:10:24.728804421Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x664b54ecae0>: 
    failed to create HCP cluster np-dg-minor-v6wk6c: failed waiting for cluster="np-dg-minor-v6wk6c" in resourcegroup="rg-np-dg-minor-v6wk6c-j7kpgqfkx4ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62536149-77a5-4b5d-a914-13bf7d06200d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62536149-77a5-4b5d-a914-13bf7d06200d",
      "name": "62536149-77a5-4b5d-a914-13bf7d06200d",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:24.096113734Z",
      "endTime": "2026-10-07T12:10:24.728804421Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x2f70e02c4940>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-z6z886s5vlhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/80ec89c1-e1d7-4d1c-9156-2ef5cd15b42c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/80ec89c1-e1d7-4d1c-9156-2ef5cd15b42c",
      "name": "80ec89c1-e1d7-4d1c-9156-2ef5cd15b42c",
      "status": "Failed",
      "startTime": "2026-10-07T11:38:48.428505117Z",
      "endTime": "2026-10-07T12:07:54.803165518Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x2f70e02c4940>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-z6z886s5vlhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/80ec89c1-e1d7-4d1c-9156-2ef5cd15b42c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/80ec89c1-e1d7-4d1c-9156-2ef5cd15b42c",
      "name": "80ec89c1-e1d7-4d1c-9156-2ef5cd15b42c",
      "status": "Failed",
      "startTime": "2026-10-07T11:38:48.428505117Z",
      "endTime": "2026-10-07T12:07:54.803165518Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-8s78nq" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2b6789bd29a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-8s78nq: failed waiting for cluster="cp-ystream-upgrade-4-23-8s78nq" in resourcegroup="rg-cp-ystream-upgrade-4-23-dlp6qsnjnzcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e17afad-249e-411b-9b0c-f5c6396456c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e17afad-249e-411b-9b0c-f5c6396456c6",
      "name": "5e17afad-249e-411b-9b0c-f5c6396456c6",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:32.119485946Z",
      "endTime": "2026-10-07T11:15:34.955895434Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-8s78nq" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2b6789bd29a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-8s78nq: failed waiting for cluster="cp-ystream-upgrade-4-23-8s78nq" in resourcegroup="rg-cp-ystream-upgrade-4-23-dlp6qsnjnzcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e17afad-249e-411b-9b0c-f5c6396456c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e17afad-249e-411b-9b0c-f5c6396456c6",
      "name": "5e17afad-249e-411b-9b0c-f5c6396456c6",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:32.119485946Z",
      "endTime": "2026-10-07T11:15:34.955895434Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x5ad8ac14f20>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qzkbcmm8mcxw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d7edca57-ba27-4ad3-8604-bffa592f9a57
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d7edca57-ba27-4ad3-8604-bffa592f9a57",
      "name": "d7edca57-ba27-4ad3-8604-bffa592f9a57",
      "status": "Failed",
      "startTime": "2026-10-07T10:47:17.981245925Z",
      "endTime": "2026-10-07T11:16:24.854771823Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x5ad8ac14f20>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qzkbcmm8mcxw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d7edca57-ba27-4ad3-8604-bffa592f9a57
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d7edca57-ba27-4ad3-8604-bffa592f9a57",
      "name": "d7edca57-ba27-4ad3-8604-bffa592f9a57",
      "status": "Failed",
      "startTime": "2026-10-07T10:47:17.981245925Z",
      "endTime": "2026-10-07T11:16:24.854771823Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-6n5rz7" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x1d92db70c300>: 
    failed to create HCP cluster cluster-zstream-4-21-6n5rz7: failed waiting for cluster="cluster-zstream-4-21-6n5rz7" in resourcegroup="rg-zstream-upgrade-4-21-2tzcc6bmrz5d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c2c70dd-1f90-4d59-8d60-3f09ac473116
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c2c70dd-1f90-4d59-8d60-3f09ac473116",
      "name": "9c2c70dd-1f90-4d59-8d60-3f09ac473116",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:52.836132625Z",
      "endTime": "2026-10-07T11:13:54.943163158Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-6n5rz7" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x1d92db70c300>: 
    failed to create HCP cluster cluster-zstream-4-21-6n5rz7: failed waiting for cluster="cluster-zstream-4-21-6n5rz7" in resourcegroup="rg-zstream-upgrade-4-21-2tzcc6bmrz5d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c2c70dd-1f90-4d59-8d60-3f09ac473116
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c2c70dd-1f90-4d59-8d60-3f09ac473116",
      "name": "9c2c70dd-1f90-4d59-8d60-3f09ac473116",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:52.836132625Z",
      "endTime": "2026-10-07T11:13:54.943163158Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x2c0664c8ea20>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-vf9txtzbnpnn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/42419f25-0a2f-4391-9457-e8a913e2ad07
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/42419f25-0a2f-4391-9457-e8a913e2ad07",
      "name": "42419f25-0a2f-4391-9457-e8a913e2ad07",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:42.287174236Z",
      "endTime": "2026-10-07T12:10:44.727935031Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x2c0664c8ea20>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-vf9txtzbnpnn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/42419f25-0a2f-4391-9457-e8a913e2ad07
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/42419f25-0a2f-4391-9457-e8a913e2ad07",
      "name": "42419f25-0a2f-4391-9457-e8a913e2ad07",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:42.287174236Z",
      "endTime": "2026-10-07T12:10:44.727935031Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xd1ce0d4a820>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-j4vrdqt97r78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f0d846d-be78-4858-ac71-4df870ff7135
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f0d846d-be78-4858-ac71-4df870ff7135",
      "name": "0f0d846d-be78-4858-ac71-4df870ff7135",
      "status": "Failed",
      "startTime": "2026-10-07T09:49:05.191507353Z",
      "endTime": "2026-10-07T10:18:14.59940059Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xd1ce0d4a820>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-j4vrdqt97r78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f0d846d-be78-4858-ac71-4df870ff7135
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f0d846d-be78-4858-ac71-4df870ff7135",
      "name": "0f0d846d-be78-4858-ac71-4df870ff7135",
      "status": "Failed",
      "startTime": "2026-10-07T09:49:05.191507353Z",
      "endTime": "2026-10-07T10:18:14.59940059Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-gwxd67" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x3d69a88ca960>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-gwxd67: failed waiting for cluster="cp-ystream-upgrade-5-1-gwxd67" in resourcegroup="rg-cp-ystream-upgrade-5-1-sz9hmhntp9kj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f8ab362-9e28-46d5-b04d-5b54432b711b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f8ab362-9e28-46d5-b04d-5b54432b711b",
      "name": "6f8ab362-9e28-46d5-b04d-5b54432b711b",
      "status": "Failed",
      "startTime": "2026-10-07T10:47:03.647392913Z",
      "endTime": "2026-10-07T11:16:04.847770934Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-gwxd67" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x3d69a88ca960>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-gwxd67: failed waiting for cluster="cp-ystream-upgrade-5-1-gwxd67" in resourcegroup="rg-cp-ystream-upgrade-5-1-sz9hmhntp9kj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f8ab362-9e28-46d5-b04d-5b54432b711b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f8ab362-9e28-46d5-b04d-5b54432b711b",
      "name": "6f8ab362-9e28-46d5-b04d-5b54432b711b",
      "status": "Failed",
      "startTime": "2026-10-07T10:47:03.647392913Z",
      "endTime": "2026-10-07T11:16:04.847770934Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-xl9nkn" with version 4.22.16 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x168a94b865c0>: 
    failed to create HCP cluster cluster-zstream-4-22-xl9nkn: failed waiting for cluster="cluster-zstream-4-22-xl9nkn" in resourcegroup="rg-zstream-upgrade-4-22-fhf2rq9bfzcg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7de41487-4bc1-4310-b1a9-cd8693231f1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7de41487-4bc1-4310-b1a9-cd8693231f1a",
      "name": "7de41487-4bc1-4310-b1a9-cd8693231f1a",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:46.782260142Z",
      "endTime": "2026-10-07T11:13:54.851175708Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-xl9nkn" with version 4.22.16 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x168a94b865c0>: 
    failed to create HCP cluster cluster-zstream-4-22-xl9nkn: failed waiting for cluster="cluster-zstream-4-22-xl9nkn" in resourcegroup="rg-zstream-upgrade-4-22-fhf2rq9bfzcg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7de41487-4bc1-4310-b1a9-cd8693231f1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7de41487-4bc1-4310-b1a9-cd8693231f1a",
      "name": "7de41487-4bc1-4310-b1a9-cd8693231f1a",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:46.782260142Z",
      "endTime": "2026-10-07T11:13:54.851175708Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0x1d4613307440>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-xccktj7qbpzs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0x1d4613307440>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-xccktj7qbpzs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-6bdqbx" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x197b95cac700>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-6bdqbx: failed waiting for cluster="cp-ystream-upgrade-5-0-6bdqbx" in resourcegroup="rg-cp-ystream-upgrade-5-0-n9sd4zrjt4nl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ce7ba26b-badb-435f-89f2-c66478a228c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ce7ba26b-badb-435f-89f2-c66478a228c2",
      "name": "ce7ba26b-badb-435f-89f2-c66478a228c2",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:45.340468133Z",
      "endTime": "2026-10-07T11:15:54.827855172Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-6bdqbx" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x197b95cac700>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-6bdqbx: failed waiting for cluster="cp-ystream-upgrade-5-0-6bdqbx" in resourcegroup="rg-cp-ystream-upgrade-5-0-n9sd4zrjt4nl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ce7ba26b-badb-435f-89f2-c66478a228c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ce7ba26b-badb-435f-89f2-c66478a228c2",
      "name": "ce7ba26b-badb-435f-89f2-c66478a228c2",
      "status": "Failed",
      "startTime": "2026-10-07T10:46:45.340468133Z",
      "endTime": "2026-10-07T11:15:54.827855172Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x22fee450ece0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-v6kzg6nzqfwv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-v6kzg6nzqfwv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x22fee450ece0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-v6kzg6nzqfwv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-v6kzg6nzqfwv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x22fba5918600>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-4lbxg4t4t5gh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c801c4e-0016-44b0-adb5-e97e7115ba1f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c801c4e-0016-44b0-adb5-e97e7115ba1f",
      "name": "7c801c4e-0016-44b0-adb5-e97e7115ba1f",
      "status": "Failed",
      "startTime": "2026-10-07T10:50:15.185124964Z",
      "endTime": "2026-10-07T11:19:24.850746241Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x22fba5918600>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-4lbxg4t4t5gh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c801c4e-0016-44b0-adb5-e97e7115ba1f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c801c4e-0016-44b0-adb5-e97e7115ba1f",
      "name": "7c801c4e-0016-44b0-adb5-e97e7115ba1f",
      "status": "Failed",
      "startTime": "2026-10-07T10:50:15.185124964Z",
      "endTime": "2026-10-07T11:19:24.850746241Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-dzdf5z" with version 5.0.0-rc.4 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2ec1ed438600>: 
    failed to create HCP cluster cluster-zstream-5-0-dzdf5z: failed waiting for cluster="cluster-zstream-5-0-dzdf5z" in resourcegroup="rg-zstream-upgrade-5-0-fvdn76jhpwwl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1857bc40-fec4-4c28-a63e-44dc6ba3733e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1857bc40-fec4-4c28-a63e-44dc6ba3733e",
      "name": "1857bc40-fec4-4c28-a63e-44dc6ba3733e",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:54.381402613Z",
      "endTime": "2026-10-07T11:13:54.951616064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-dzdf5z" with version 5.0.0-rc.4 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2ec1ed438600>: 
    failed to create HCP cluster cluster-zstream-5-0-dzdf5z: failed waiting for cluster="cluster-zstream-5-0-dzdf5z" in resourcegroup="rg-zstream-upgrade-5-0-fvdn76jhpwwl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1857bc40-fec4-4c28-a63e-44dc6ba3733e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1857bc40-fec4-4c28-a63e-44dc6ba3733e",
      "name": "1857bc40-fec4-4c28-a63e-44dc6ba3733e",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:54.381402613Z",
      "endTime": "2026-10-07T11:13:54.951616064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-hstn44bjct9p/cluster-candidate-4-23-b4dmlz should provision
Unexpected error:
    <*fmt.wrapError | 0x3d5329548a0>: 
    failed to create HCP cluster cluster-candidate-4-23-b4dmlz: failed waiting for cluster="cluster-candidate-4-23-b4dmlz" in resourcegroup="rg-candidate-4-23-hstn44bjct9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3cf4f402-ca73-41e8-bc7a-b95fe95ddbb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3cf4f402-ca73-41e8-bc7a-b95fe95ddbb3",
      "name": "3cf4f402-ca73-41e8-bc7a-b95fe95ddbb3",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:15.500215383Z",
      "endTime": "2026-10-07T11:13:24.832957555Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-hstn44bjct9p/cluster-candidate-4-23-b4dmlz should provision
Unexpected error:
    <*fmt.wrapError | 0x3d5329548a0>: 
    failed to create HCP cluster cluster-candidate-4-23-b4dmlz: failed waiting for cluster="cluster-candidate-4-23-b4dmlz" in resourcegroup="rg-candidate-4-23-hstn44bjct9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3cf4f402-ca73-41e8-bc7a-b95fe95ddbb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3cf4f402-ca73-41e8-bc7a-b95fe95ddbb3",
      "name": "3cf4f402-ca73-41e8-bc7a-b95fe95ddbb3",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:15.500215383Z",
      "endTime": "2026-10-07T11:13:24.832957555Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xd878dbfc8e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-rtvvfx44j8ld": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-rtvvfx44j8ld/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xd878dbfc8e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-rtvvfx44j8ld": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-rtvvfx44j8ld/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-cl6plzw4cxhg/cluster-candidate-4-20-ptf54w should provision
Unexpected error:
    <*fmt.wrapError | 0x3294f0080980>: 
    failed to create HCP cluster cluster-candidate-4-20-ptf54w: failed waiting for cluster="cluster-candidate-4-20-ptf54w" in resourcegroup="rg-candidate-4-20-cl6plzw4cxhg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/246ff8c9-d11b-4a89-906d-6858801ce1bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/246ff8c9-d11b-4a89-906d-6858801ce1bc",
      "name": "246ff8c9-d11b-4a89-906d-6858801ce1bc",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:12.829533656Z",
      "endTime": "2026-10-07T10:21:14.528988109Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-cl6plzw4cxhg/cluster-candidate-4-20-ptf54w should provision
Unexpected error:
    <*fmt.wrapError | 0x3294f0080980>: 
    failed to create HCP cluster cluster-candidate-4-20-ptf54w: failed waiting for cluster="cluster-candidate-4-20-ptf54w" in resourcegroup="rg-candidate-4-20-cl6plzw4cxhg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/246ff8c9-d11b-4a89-906d-6858801ce1bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/246ff8c9-d11b-4a89-906d-6858801ce1bc",
      "name": "246ff8c9-d11b-4a89-906d-6858801ce1bc",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:12.829533656Z",
      "endTime": "2026-10-07T10:21:14.528988109Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x3723bf130ca0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-vdnbdccqmbl9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-vdnbdccqmbl9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x3723bf130ca0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-vdnbdccqmbl9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-vdnbdccqmbl9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2669c8834500>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-hsv7wr49qksv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f18e0c20-bbd8-41c2-9e07-ac4d066319fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f18e0c20-bbd8-41c2-9e07-ac4d066319fa",
      "name": "f18e0c20-bbd8-41c2-9e07-ac4d066319fa",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:56.465322587Z",
      "endTime": "2026-10-07T10:18:04.504330062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2669c8834500>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-hsv7wr49qksv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f18e0c20-bbd8-41c2-9e07-ac4d066319fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f18e0c20-bbd8-41c2-9e07-ac4d066319fa",
      "name": "f18e0c20-bbd8-41c2-9e07-ac4d066319fa",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:56.465322587Z",
      "endTime": "2026-10-07T10:18:04.504330062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x162888dc49e0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-bp2mcv6f588h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a9746092-ea51-4e95-908b-677b0574f350
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a9746092-ea51-4e95-908b-677b0574f350",
      "name": "a9746092-ea51-4e95-908b-677b0574f350",
      "status": "Failed",
      "startTime": "2026-10-07T09:50:56.187197768Z",
      "endTime": "2026-10-07T10:20:04.504909256Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x162888dc49e0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-bp2mcv6f588h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a9746092-ea51-4e95-908b-677b0574f350
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a9746092-ea51-4e95-908b-677b0574f350",
      "name": "a9746092-ea51-4e95-908b-677b0574f350",
      "status": "Failed",
      "startTime": "2026-10-07T09:50:56.187197768Z",
      "endTime": "2026-10-07T10:20:04.504909256Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-wjlj9cf4zcnw/cluster-candidate-5-1-k5brcg should provision
Unexpected error:
    <*fmt.wrapError | 0x2c0270464680>: 
    failed to create HCP cluster cluster-candidate-5-1-k5brcg: failed waiting for cluster="cluster-candidate-5-1-k5brcg" in resourcegroup="rg-candidate-5-1-wjlj9cf4zcnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f59e7e7-9129-4701-b675-75127fcdc2eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f59e7e7-9129-4701-b675-75127fcdc2eb",
      "name": "9f59e7e7-9129-4701-b675-75127fcdc2eb",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:26.380363687Z",
      "endTime": "2026-10-07T11:13:34.962794383Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-wjlj9cf4zcnw/cluster-candidate-5-1-k5brcg should provision
Unexpected error:
    <*fmt.wrapError | 0x2c0270464680>: 
    failed to create HCP cluster cluster-candidate-5-1-k5brcg: failed waiting for cluster="cluster-candidate-5-1-k5brcg" in resourcegroup="rg-candidate-5-1-wjlj9cf4zcnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f59e7e7-9129-4701-b675-75127fcdc2eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9f59e7e7-9129-4701-b675-75127fcdc2eb",
      "name": "9f59e7e7-9129-4701-b675-75127fcdc2eb",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:26.380363687Z",
      "endTime": "2026-10-07T11:13:34.962794383Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3fd68dad0ac0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6mv68nffvl22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd123085-ce38-4a8b-8b84-938bfcfb8b88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd123085-ce38-4a8b-8b84-938bfcfb8b88",
      "name": "cd123085-ce38-4a8b-8b84-938bfcfb8b88",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:56.476908409Z",
      "endTime": "2026-10-07T10:18:04.678303243Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3fd68dad0ac0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6mv68nffvl22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd123085-ce38-4a8b-8b84-938bfcfb8b88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd123085-ce38-4a8b-8b84-938bfcfb8b88",
      "name": "cd123085-ce38-4a8b-8b84-938bfcfb8b88",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:56.476908409Z",
      "endTime": "2026-10-07T10:18:04.678303243Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-f8hx5m with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x37308114340>: 
    failed to create HCP cluster np-version-upgrade-cluster-f8hx5m: failed waiting for cluster="np-version-upgrade-cluster-f8hx5m" in resourcegroup="rg-np-version-upgrade-f8hx5m-8jdlnlxvfr8j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f71f735-23a2-44d3-8bee-778955a7585f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f71f735-23a2-44d3-8bee-778955a7585f",
      "name": "1f71f735-23a2-44d3-8bee-778955a7585f",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:50.690197671Z",
      "endTime": "2026-10-07T12:08:54.942764093Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-f8hx5m with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x37308114340>: 
    failed to create HCP cluster np-version-upgrade-cluster-f8hx5m: failed waiting for cluster="np-version-upgrade-cluster-f8hx5m" in resourcegroup="rg-np-version-upgrade-f8hx5m-8jdlnlxvfr8j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f71f735-23a2-44d3-8bee-778955a7585f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f71f735-23a2-44d3-8bee-778955a7585f",
      "name": "1f71f735-23a2-44d3-8bee-778955a7585f",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:50.690197671Z",
      "endTime": "2026-10-07T12:08:54.942764093Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x1a50070451c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-8fphbzx94qxr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/84804da6-d82b-4a54-acff-55763e6c56e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/84804da6-d82b-4a54-acff-55763e6c56e3",
      "name": "84804da6-d82b-4a54-acff-55763e6c56e3",
      "status": "Failed",
      "startTime": "2026-10-07T09:50:58.529983615Z",
      "endTime": "2026-10-07T10:20:04.629346768Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x1a50070451c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-8fphbzx94qxr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/84804da6-d82b-4a54-acff-55763e6c56e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/84804da6-d82b-4a54-acff-55763e6c56e3",
      "name": "84804da6-d82b-4a54-acff-55763e6c56e3",
      "status": "Failed",
      "startTime": "2026-10-07T09:50:58.529983615Z",
      "endTime": "2026-10-07T10:20:04.629346768Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xbae29765720>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-z5fwltqgzc4v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5482cdd-de26-4821-85d9-d6c90378d1e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5482cdd-de26-4821-85d9-d6c90378d1e9",
      "name": "c5482cdd-de26-4821-85d9-d6c90378d1e9",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:08.573668314Z",
      "endTime": "2026-10-07T10:21:14.503850093Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xbae29765720>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-z5fwltqgzc4v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5482cdd-de26-4821-85d9-d6c90378d1e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c5482cdd-de26-4821-85d9-d6c90378d1e9",
      "name": "c5482cdd-de26-4821-85d9-d6c90378d1e9",
      "status": "Failed",
      "startTime": "2026-10-07T09:52:08.573668314Z",
      "endTime": "2026-10-07T10:21:14.503850093Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
09:23:41 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:23:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x2fbb237581e0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kc94rh5nhxb8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kc94rh5nhxb8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2fbb237ab310>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x2fbb237581e0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kc94rh5nhxb8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kc94rh5nhxb8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2fbb237ab310>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x3fbdfa206980>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7zh6fp8fgksk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5147c191-3d75-411b-b997-ea46e2d7cb21
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5147c191-3d75-411b-b997-ea46e2d7cb21",
      "name": "5147c191-3d75-411b-b997-ea46e2d7cb21",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:07.233759701Z",
      "endTime": "2026-10-07T12:10:14.721254823Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x3fbdfa206980>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7zh6fp8fgksk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5147c191-3d75-411b-b997-ea46e2d7cb21
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5147c191-3d75-411b-b997-ea46e2d7cb21",
      "name": "5147c191-3d75-411b-b997-ea46e2d7cb21",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:07.233759701Z",
      "endTime": "2026-10-07T12:10:14.721254823Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3c820c72ab60>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-dlvtfxrd2j7s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a0b16c5f-b10f-427f-9f76-5f47695c8e2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a0b16c5f-b10f-427f-9f76-5f47695c8e2f",
      "name": "a0b16c5f-b10f-427f-9f76-5f47695c8e2f",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:50.969969751Z",
      "endTime": "2026-10-07T12:10:54.729673473Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3c820c72ab60>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-dlvtfxrd2j7s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a0b16c5f-b10f-427f-9f76-5f47695c8e2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a0b16c5f-b10f-427f-9f76-5f47695c8e2f",
      "name": "a0b16c5f-b10f-427f-9f76-5f47695c8e2f",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:50.969969751Z",
      "endTime": "2026-10-07T12:10:54.729673473Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-nwj77t" with version 4.20.40 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x275bdca84120>: 
    failed to create HCP cluster cluster-zstream-4-20-nwj77t: failed waiting for cluster="cluster-zstream-4-20-nwj77t" in resourcegroup="rg-zstream-upgrade-4-20-szq75nd4jwdq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4a7a5069-7fe3-4404-a6ab-059ffdfd5c85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4a7a5069-7fe3-4404-a6ab-059ffdfd5c85",
      "name": "4a7a5069-7fe3-4404-a6ab-059ffdfd5c85",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:35.717450122Z",
      "endTime": "2026-10-07T11:13:44.953502768Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-nwj77t" with version 4.20.40 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x275bdca84120>: 
    failed to create HCP cluster cluster-zstream-4-20-nwj77t: failed waiting for cluster="cluster-zstream-4-20-nwj77t" in resourcegroup="rg-zstream-upgrade-4-20-szq75nd4jwdq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4a7a5069-7fe3-4404-a6ab-059ffdfd5c85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4a7a5069-7fe3-4404-a6ab-059ffdfd5c85",
      "name": "4a7a5069-7fe3-4404-a6ab-059ffdfd5c85",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:35.717450122Z",
      "endTime": "2026-10-07T11:13:44.953502768Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
09:23:41 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x13ce01f6d240>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-lf5wbzx9gtnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7376de8f-61ce-423e-aa70-1db4f06f54df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7376de8f-61ce-423e-aa70-1db4f06f54df",
      "name": "7376de8f-61ce-423e-aa70-1db4f06f54df",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:54.383796786Z",
      "endTime": "2026-10-07T10:17:54.50128014Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x13ce01f6d240>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-lf5wbzx9gtnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7376de8f-61ce-423e-aa70-1db4f06f54df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7376de8f-61ce-423e-aa70-1db4f06f54df",
      "name": "7376de8f-61ce-423e-aa70-1db4f06f54df",
      "status": "Failed",
      "startTime": "2026-10-07T09:48:54.383796786Z",
      "endTime": "2026-10-07T10:17:54.50128014Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 24m 47s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast e2e failed n/a 33
e2e (33)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x2191d5b1d3e0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-5wqhb2" resourcegroup="rg-np-skip-minor-5wqhb2-pcpb5hkjlxqk": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-5wqhb2-pcpb5hkjlxqk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-5wqhb2/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x2191d5b1d3e0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-5wqhb2" resourcegroup="rg-np-skip-minor-5wqhb2-pcpb5hkjlxqk": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-5wqhb2-pcpb5hkjlxqk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-5wqhb2/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5g7cxp" for cluster "cluster-candidate-4-22-5g7cxp"
Unexpected error:
    <*fmt.wrapError | 0x2051b5cf8420>: 
    failed to create NodePool np-5g7cxp: failed waiting for nodepool="np-5g7cxp" for cluster "cluster-candidate-4-22-5g7cxp" in resourcegroup="rg-candidate-4-22-4vkh858b4p5m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/2078827d-3863-4f50-abad-4b08b8a19530
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/2078827d-3863-4f50-abad-4b08b8a19530",
      "name": "2078827d-3863-4f50-abad-4b08b8a19530",
      "status": "Failed",
      "startTime": "2026-10-07T02:49:22.855228495Z",
      "endTime": "2026-10-07T03:18:22.99461352Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb to exist\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb to exist\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5g7cxp" for cluster "cluster-candidate-4-22-5g7cxp"
Unexpected error:
    <*fmt.wrapError | 0x2051b5cf8420>: 
    failed to create NodePool np-5g7cxp: failed waiting for nodepool="np-5g7cxp" for cluster "cluster-candidate-4-22-5g7cxp" in resourcegroup="rg-candidate-4-22-4vkh858b4p5m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/2078827d-3863-4f50-abad-4b08b8a19530
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/2078827d-3863-4f50-abad-4b08b8a19530",
      "name": "2078827d-3863-4f50-abad-4b08b8a19530",
      "status": "Failed",
      "startTime": "2026-10-07T02:49:22.855228495Z",
      "endTime": "2026-10-07T03:18:22.99461352Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb to exist\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-7nxzb to exist\nMachine r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-4vkh858b4p5m-candidate-5g7cxp--managed/providers/Microsoft.Compute/virtualMachines/r3p7j3d3k1v6n6z-np-5g7cxp-kkrh8-bdpdz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
02:00:56 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x266375a9e030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rsx69mvsw2xq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/86fb5972-fae9-42a5-b2cc-cef72453d35a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/86fb5972-fae9-42a5-b2cc-cef72453d35a",
      "name": "86fb5972-fae9-42a5-b2cc-cef72453d35a",
      "status": "Failed",
      "startTime": "2026-10-07T02:19:24.17651873Z",
      "endTime": "2026-10-07T02:48:31.262778869Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-24jdj-5chf2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-5chf2 to exist\nMachine cilium-cluster-cilium-np-24jdj-dvggb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-dvggb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-24jdj-5chf2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-5chf2 to exist\nMachine cilium-cluster-cilium-np-24jdj-dvggb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-dvggb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x266375a9e030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rsx69mvsw2xq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/86fb5972-fae9-42a5-b2cc-cef72453d35a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/86fb5972-fae9-42a5-b2cc-cef72453d35a",
      "name": "86fb5972-fae9-42a5-b2cc-cef72453d35a",
      "status": "Failed",
      "startTime": "2026-10-07T02:19:24.17651873Z",
      "endTime": "2026-10-07T02:48:31.262778869Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-24jdj-5chf2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-5chf2 to exist\nMachine cilium-cluster-cilium-np-24jdj-dvggb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-dvggb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-24jdj-5chf2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-5chf2 to exist\nMachine cilium-cluster-cilium-np-24jdj-dvggb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rsx69mvsw2xq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-24jdj-dvggb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
02:00:56 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3b65283c5780>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qcwx6cjhvdwl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/75bd1e92-642d-44cc-9f1d-7a0b14555639
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/75bd1e92-642d-44cc-9f1d-7a0b14555639",
      "name": "75bd1e92-642d-44cc-9f1d-7a0b14555639",
      "status": "Failed",
      "startTime": "2026-10-07T03:30:42.032719201Z",
      "endTime": "2026-10-07T03:59:47.578313034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf to exist\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf to exist\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3b65283c5780>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qcwx6cjhvdwl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/75bd1e92-642d-44cc-9f1d-7a0b14555639
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/75bd1e92-642d-44cc-9f1d-7a0b14555639",
      "name": "75bd1e92-642d-44cc-9f1d-7a0b14555639",
      "status": "Failed",
      "startTime": "2026-10-07T03:30:42.032719201Z",
      "endTime": "2026-10-07T03:59:47.578313034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf to exist\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-96sxf to exist\nMachine g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qcwx6cjhvdwl--managed/providers/Microsoft.Compute/virtualMachines/g3y2u3u9b3i0a3e-np-delete-main-mk65h-hb6kz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x2b54f9373160>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-rb5sd7" resourcegroup="rg-np-noedge-rb5sd7-m8nqd7x28f72": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-rb5sd7-m8nqd7x28f72/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-rb5sd7/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x2b54f9373160>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-rb5sd7" resourcegroup="rg-np-noedge-rb5sd7-m8nqd7x28f72": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-rb5sd7-m8nqd7x28f72/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-rb5sd7/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x2846e9590e40>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-6gjzds" resourcegroup="rg-np-downgrade-6gjzds-86mbqsnm8drv": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-6gjzds-86mbqsnm8drv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-6gjzds/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x2846e9590e40>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-6gjzds" resourcegroup="rg-np-downgrade-6gjzds-86mbqsnm8drv": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-6gjzds-86mbqsnm8drv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-6gjzds/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x231d4b99bc00>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lxpkjd" resourcegroup="rg-np-version-upgrade-lxpkjd-r2d8ctn9k9gz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lxpkjd-r2d8ctn9k9gz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lxpkjd/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x231d4b99bc00>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lxpkjd" resourcegroup="rg-np-version-upgrade-lxpkjd-r2d8ctn9k9gz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lxpkjd-r2d8ctn9k9gz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lxpkjd/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-8l7dvc" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x3cfbe8935230>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-xjbz46kpx22f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-8l7dvc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-8l7dvc" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x3cfbe8935230>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-xjbz46kpx22f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-8l7dvc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-t6wdwh" for cluster "cluster-candidate-5-1-t6wdwh"
Unexpected error:
    <*fmt.wrapError | 0x3dd08fa06500>: 
    failed to create NodePool np-t6wdwh: failed waiting for nodepool="np-t6wdwh" for cluster "cluster-candidate-5-1-t6wdwh" in resourcegroup="rg-candidate-5-1-kfz288sb9rh2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1d57d7e0-5573-4196-8ff1-f505d1b163e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1d57d7e0-5573-4196-8ff1-f505d1b163e5",
      "name": "1d57d7e0-5573-4196-8ff1-f505d1b163e5",
      "status": "Failed",
      "startTime": "2026-10-07T02:52:22.077189924Z",
      "endTime": "2026-10-07T03:21:23.162976544Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l to exist\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l to exist\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-t6wdwh" for cluster "cluster-candidate-5-1-t6wdwh"
Unexpected error:
    <*fmt.wrapError | 0x3dd08fa06500>: 
    failed to create NodePool np-t6wdwh: failed waiting for nodepool="np-t6wdwh" for cluster "cluster-candidate-5-1-t6wdwh" in resourcegroup="rg-candidate-5-1-kfz288sb9rh2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1d57d7e0-5573-4196-8ff1-f505d1b163e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1d57d7e0-5573-4196-8ff1-f505d1b163e5",
      "name": "1d57d7e0-5573-4196-8ff1-f505d1b163e5",
      "status": "Failed",
      "startTime": "2026-10-07T02:52:22.077189924Z",
      "endTime": "2026-10-07T03:21:23.162976544Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l to exist\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-6w44l to exist\nMachine u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kfz288sb9rh2-candidate-t6wdwh--managed/providers/Microsoft.Compute/virtualMachines/u3e6h1a2m5i1o7s-np-t6wdwh-92gd4-wd24r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x316f2960af60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5wjtv7scf56q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x316f2960af60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5wjtv7scf56q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mfcgvm" for cluster "cluster-candidate-4-20-mfcgvm"
Unexpected error:
    <*fmt.wrapError | 0x5d26d82a420>: 
    failed to create NodePool np-mfcgvm: failed waiting for nodepool="np-mfcgvm" for cluster "cluster-candidate-4-20-mfcgvm" in resourcegroup="rg-candidate-4-20-lh5ndcqmnzxh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c538dcdd-16ed-410d-8083-51bdf151f95d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c538dcdd-16ed-410d-8083-51bdf151f95d",
      "name": "c538dcdd-16ed-410d-8083-51bdf151f95d",
      "status": "Failed",
      "startTime": "2026-10-07T02:50:54.360465912Z",
      "endTime": "2026-10-07T03:20:03.12214784Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf to exist\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf to exist\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mfcgvm" for cluster "cluster-candidate-4-20-mfcgvm"
Unexpected error:
    <*fmt.wrapError | 0x5d26d82a420>: 
    failed to create NodePool np-mfcgvm: failed waiting for nodepool="np-mfcgvm" for cluster "cluster-candidate-4-20-mfcgvm" in resourcegroup="rg-candidate-4-20-lh5ndcqmnzxh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c538dcdd-16ed-410d-8083-51bdf151f95d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c538dcdd-16ed-410d-8083-51bdf151f95d",
      "name": "c538dcdd-16ed-410d-8083-51bdf151f95d",
      "status": "Failed",
      "startTime": "2026-10-07T02:50:54.360465912Z",
      "endTime": "2026-10-07T03:20:03.12214784Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf to exist\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-lkcwf to exist\nMachine g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lh5ndcqmnzxh-candidate-mfcgvm--managed/providers/Microsoft.Compute/virtualMachines/g5c8w8d3q7e5i9l-np-mfcgvm-64sl9-qmwtz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x37d6f372b6a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-8sc8h7sjfdcv": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-8sc8h7sjfdcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x37d6f372b6a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-8sc8h7sjfdcv": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-8sc8h7sjfdcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-p4lgwf" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x1b0947b50a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-2gpxkvcjbvk8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-p4lgwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-p4lgwf" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x1b0947b50a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-2gpxkvcjbvk8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-p4lgwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3dec911a2048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-54gvlbbq8fjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c7f98f33-b298-4d0a-a913-93aac3ffc091
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c7f98f33-b298-4d0a-a913-93aac3ffc091",
      "name": "c7f98f33-b298-4d0a-a913-93aac3ffc091",
      "status": "Failed",
      "startTime": "2026-10-07T02:20:55.959277742Z",
      "endTime": "2026-10-07T02:50:00.936442552Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-d5fls-gl48s: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-d5fls-m9lj5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-54gvlbbq8fjw--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-d5fls-m9lj5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-d5fls-gl48s: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-d5fls-m9lj5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-54gvlbbq8fjw--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-d5fls-m9lj5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3dec911a2048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-54gvlbbq8fjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c7f98f33-b298-4d0a-a913-93aac3ffc091
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c7f98f33-b298-4d0a-a913-93aac3ffc091",
      "name": "c7f98f33-b298-4d0a-a913-93aac3ffc091",
      "status": "Failed",
      "startTime": "2026-10-07T02:20:55.959277742Z",
      "endTime": "2026-10-07T02:50:00.936442552Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-d5fls-gl48s: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-d5fls-m9lj5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-54gvlbbq8fjw--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-d5fls-m9lj5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-d5fls-gl48s: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-d5fls-m9lj5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-54gvlbbq8fjw--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-d5fls-m9lj5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-82pv4r" for cluster "cluster-candidate-4-23-82pv4r"
Unexpected error:
    <*fmt.wrapError | 0x33dd9953ad20>: 
    failed to create NodePool np-82pv4r: failed waiting for nodepool="np-82pv4r" for cluster "cluster-candidate-4-23-82pv4r" in resourcegroup="rg-candidate-4-23-6g4s6hnh7twn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/763f07a9-7f84-4d35-b440-501db428cd35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/763f07a9-7f84-4d35-b440-501db428cd35",
      "name": "763f07a9-7f84-4d35-b440-501db428cd35",
      "status": "Failed",
      "startTime": "2026-10-07T02:49:16.715244148Z",
      "endTime": "2026-10-07T03:18:22.978126797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96 to exist\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96 to exist\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-82pv4r" for cluster "cluster-candidate-4-23-82pv4r"
Unexpected error:
    <*fmt.wrapError | 0x33dd9953ad20>: 
    failed to create NodePool np-82pv4r: failed waiting for nodepool="np-82pv4r" for cluster "cluster-candidate-4-23-82pv4r" in resourcegroup="rg-candidate-4-23-6g4s6hnh7twn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/763f07a9-7f84-4d35-b440-501db428cd35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/763f07a9-7f84-4d35-b440-501db428cd35",
      "name": "763f07a9-7f84-4d35-b440-501db428cd35",
      "status": "Failed",
      "startTime": "2026-10-07T02:49:16.715244148Z",
      "endTime": "2026-10-07T03:18:22.978126797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96 to exist\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-ffv96 to exist\nMachine b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6g4s6hnh7twn-candidate-82pv4r--managed/providers/Microsoft.Compute/virtualMachines/b9q6c1i1o4c2m6f-np-82pv4r-tcwl6-nncb2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-wfqdx6" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x35670f110a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-m89bbfgxxvxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-wfqdx6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-wfqdx6" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x35670f110a20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-m89bbfgxxvxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-wfqdx6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-jq9x5k" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x13bed9716540>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-lr4cgtdlbsdg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-jq9x5k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-jq9x5k" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x13bed9716540>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-lr4cgtdlbsdg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-jq9x5k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x17b599a4e4b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-rvt8dfwzw6cv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x17b599a4e4b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-rvt8dfwzw6cv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-h5kqdt" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x2688b23f290>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-dqv8pzksfg2w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-h5kqdt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-h5kqdt" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x2688b23f290>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-dqv8pzksfg2w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-h5kqdt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-gcjnnr" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x2862c3242150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-97btfzzdk5qt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gcjnnr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-gcjnnr" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x2862c3242150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-97btfzzdk5qt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gcjnnr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x7308776540>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-92bg56" resourcegroup="rg-np-version-upgrade-92bg56-8frlm96hwhs2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-92bg56-8frlm96hwhs2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-92bg56/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x7308776540>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-92bg56" resourcegroup="rg-np-version-upgrade-92bg56-8frlm96hwhs2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-92bg56-8frlm96hwhs2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-92bg56/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x2730e9c9c180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-4297wtnz7lkp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e9b6cfe3-4de4-41e6-9696-6b3171ff2374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e9b6cfe3-4de4-41e6-9696-6b3171ff2374",
      "name": "e9b6cfe3-4de4-41e6-9696-6b3171ff2374",
      "status": "Failed",
      "startTime": "2026-10-07T02:18:16.717381704Z",
      "endTime": "2026-10-07T02:47:20.922673Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-9ndmt-lqfsm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-4297wtnz7lkp--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-9ndmt-lqfsm to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-9ndmt-lqfsm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-4297wtnz7lkp--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-9ndmt-lqfsm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x2730e9c9c180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-4297wtnz7lkp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e9b6cfe3-4de4-41e6-9696-6b3171ff2374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e9b6cfe3-4de4-41e6-9696-6b3171ff2374",
      "name": "e9b6cfe3-4de4-41e6-9696-6b3171ff2374",
      "status": "Failed",
      "startTime": "2026-10-07T02:18:16.717381704Z",
      "endTime": "2026-10-07T02:47:20.922673Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-9ndmt-lqfsm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-4297wtnz7lkp--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-9ndmt-lqfsm to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-9ndmt-lqfsm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-4297wtnz7lkp--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-9ndmt-lqfsm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x313b400003a0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-mqrq8s" resourcegroup="rg-np-dg-minor-mqrq8s-zhzsjsdzbsjk": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-mqrq8s-zhzsjsdzbsjk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-mqrq8s/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x313b400003a0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-mqrq8s" resourcegroup="rg-np-dg-minor-mqrq8s-zhzsjsdzbsjk": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-mqrq8s-zhzsjsdzbsjk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-mqrq8s/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-r6bz6s" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x2ed2f27658c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-7jmjflmmkzbd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-r6bz6s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-r6bz6s" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x2ed2f27658c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-7jmjflmmkzbd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-r6bz6s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-fg9qp7" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x27965fe99170>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-bd2blzzd24m7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-fg9qp7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-fg9qp7" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x27965fe99170>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-bd2blzzd24m7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-fg9qp7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5sbtvz" for cluster "cluster-candidate-5-0-5sbtvz"
Unexpected error:
    <*fmt.wrapError | 0x238acb1486c0>: 
    failed to create NodePool np-5sbtvz: failed waiting for nodepool="np-5sbtvz" for cluster "cluster-candidate-5-0-5sbtvz" in resourcegroup="rg-candidate-5-0-vmsjtdc4dmc2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/94d7a272-7f93-42b4-87a7-e5e1f90fc4aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/94d7a272-7f93-42b4-87a7-e5e1f90fc4aa",
      "name": "94d7a272-7f93-42b4-87a7-e5e1f90fc4aa",
      "status": "Failed",
      "startTime": "2026-10-07T02:53:44.600418059Z",
      "endTime": "2026-10-07T03:22:52.95122632Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n to exist\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n to exist\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5sbtvz" for cluster "cluster-candidate-5-0-5sbtvz"
Unexpected error:
    <*fmt.wrapError | 0x238acb1486c0>: 
    failed to create NodePool np-5sbtvz: failed waiting for nodepool="np-5sbtvz" for cluster "cluster-candidate-5-0-5sbtvz" in resourcegroup="rg-candidate-5-0-vmsjtdc4dmc2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/94d7a272-7f93-42b4-87a7-e5e1f90fc4aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/94d7a272-7f93-42b4-87a7-e5e1f90fc4aa",
      "name": "94d7a272-7f93-42b4-87a7-e5e1f90fc4aa",
      "status": "Failed",
      "startTime": "2026-10-07T02:53:44.600418059Z",
      "endTime": "2026-10-07T03:22:52.95122632Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n to exist\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-h6d5n to exist\nMachine c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vmsjtdc4dmc2-candidate-5sbtvz--managed/providers/Microsoft.Compute/virtualMachines/c8c6d7q4q9v4l8v-np-5sbtvz-54sd7-vxnm5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-rdvrsz" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc9376a12f60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-q7mr47v6tpnm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rdvrsz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-rdvrsz" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc9376a12f60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-q7mr47v6tpnm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rdvrsz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-8t9fz9" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xd4a08aa9740>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-5jggbjxvkqw6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-8t9fz9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-8t9fz9" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xd4a08aa9740>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-5jggbjxvkqw6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-8t9fz9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x3b23ae9e81b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s64nkbtbvr4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x3b23ae9e81b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s64nkbtbvr4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jxhjcb" for cluster "cluster-candidate-4-21-jxhjcb"
Unexpected error:
    <*fmt.wrapError | 0x1846fa80a900>: 
    failed to create NodePool np-jxhjcb: failed waiting for nodepool="np-jxhjcb" for cluster "cluster-candidate-4-21-jxhjcb" in resourcegroup="rg-candidate-4-21-8w5wwd2jrrxw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa63c7e2-d755-41d4-a99d-595192e3cb6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa63c7e2-d755-41d4-a99d-595192e3cb6e",
      "name": "fa63c7e2-d755-41d4-a99d-595192e3cb6e",
      "status": "Failed",
      "startTime": "2026-10-07T02:50:57.300613412Z",
      "endTime": "2026-10-07T03:20:02.945126702Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg to exist\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg to exist\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jxhjcb" for cluster "cluster-candidate-4-21-jxhjcb"
Unexpected error:
    <*fmt.wrapError | 0x1846fa80a900>: 
    failed to create NodePool np-jxhjcb: failed waiting for nodepool="np-jxhjcb" for cluster "cluster-candidate-4-21-jxhjcb" in resourcegroup="rg-candidate-4-21-8w5wwd2jrrxw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa63c7e2-d755-41d4-a99d-595192e3cb6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa63c7e2-d755-41d4-a99d-595192e3cb6e",
      "name": "fa63c7e2-d755-41d4-a99d-595192e3cb6e",
      "status": "Failed",
      "startTime": "2026-10-07T02:50:57.300613412Z",
      "endTime": "2026-10-07T03:20:02.945126702Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg to exist\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-gnvwg to exist\nMachine x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8w5wwd2jrrxw-candidate-jxhjcb--managed/providers/Microsoft.Compute/virtualMachines/x1s8c4w1t6a8s2s-np-jxhjcb-mqmzf-khzns to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:56 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x516fee870a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-hlstwcddsl64" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d577455b-50b8-4e4b-85d5-72a517862832
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d577455b-50b8-4e4b-85d5-72a517862832",
      "name": "d577455b-50b8-4e4b-85d5-72a517862832",
      "status": "Failed",
      "startTime": "2026-10-07T02:22:43.989386912Z",
      "endTime": "2026-10-07T02:51:50.953880817Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5 to exist\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5 to exist\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x516fee870a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-hlstwcddsl64" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d577455b-50b8-4e4b-85d5-72a517862832
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d577455b-50b8-4e4b-85d5-72a517862832",
      "name": "d577455b-50b8-4e4b-85d5-72a517862832",
      "status": "Failed",
      "startTime": "2026-10-07T02:22:43.989386912Z",
      "endTime": "2026-10-07T02:51:50.953880817Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5 to exist\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-mpds5 to exist\nMachine l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-hlstwcddsl64--managed/providers/Microsoft.Compute/virtualMachines/l1n4n6o5d5c1t1h-np-1-j4gb2-w2c2d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x12b4a1bbd7e0>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-7k79kh" resourcegroup="rg-np-version-upgrade-7k79kh-mbzxs6n4k6hw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7k79kh-mbzxs6n4k6hw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7k79kh/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x12b4a1bbd7e0>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-7k79kh" resourcegroup="rg-np-version-upgrade-7k79kh-mbzxs6n4k6hw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7k79kh-mbzxs6n4k6hw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7k79kh/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:56 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x2e96d2061e60>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tn2zc5z4vc6x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-tn2zc5z4vc6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x2e96d2061e60>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-tn2zc5z4vc6x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-tn2zc5z4vc6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred