<
Single Day: Sep 29▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: INT

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 37m 25s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westus3 e2e failed n/a 2
e2e (2)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
21:03:38 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00134a240>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-lf58x6mrt8sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38",
      "name": "6899b842-6be7-4752-9378-45ead2f46f38",
      "status": "Failed",
      "startTime": "2026-09-29T22:48:49.695468888Z",
      "endTime": "2026-09-29T23:07:57.514241775Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00134a240>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-lf58x6mrt8sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38",
      "name": "6899b842-6be7-4752-9378-45ead2f46f38",
      "status": "Failed",
      "startTime": "2026-09-29T22:48:49.695468888Z",
      "endTime": "2026-09-29T23:07:57.514241775Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
21:03:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010311c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-zbbkqjx7fwrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f",
      "name": "15a80b6b-fe14-4fa6-a826-e605868e189f",
      "status": "Failed",
      "startTime": "2026-09-29T21:44:03.296557Z",
      "endTime": "2026-09-29T22:03:11.679917905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010311c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-zbbkqjx7fwrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f",
      "name": "15a80b6b-fe14-4fa6-a826-e605868e189f",
      "status": "Failed",
      "startTime": "2026-09-29T21:44:03.296557Z",
      "endTime": "2026-09-29T22:03:11.679917905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 25m 53s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westus3 e2e failed n/a 2
e2e (2)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
16:09:07 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f32080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-thvwrhnx9f2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "name": "aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "status": "Failed",
      "startTime": "2026-09-29T16:37:13.215287875Z",
      "endTime": "2026-09-29T16:56:16.225730373Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f32080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-thvwrhnx9f2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "name": "aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "status": "Failed",
      "startTime": "2026-09-29T16:37:13.215287875Z",
      "endTime": "2026-09-29T16:56:16.225730373Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
16:09:07 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc00143c978>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc00143c978>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
2h 28m 52s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westus3 e2e failed n/a 2
e2e (2)
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
13:34:27 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc00090a4c8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc00090a4c8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:34:27 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009bd5e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2njpnbrz5jg5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e034572f-71f9-4484-a06b-08d411aba3d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e034572f-71f9-4484-a06b-08d411aba3d9",
      "name": "e034572f-71f9-4484-a06b-08d411aba3d9",
      "status": "Failed",
      "startTime": "2026-09-29T14:28:57.448756437Z",
      "endTime": "2026-09-29T14:48:03.628370174Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n1q8y3e1s2j2m9q-np-1-spldz-2nngd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-2njpnbrz5jg5--managed/providers/Microsoft.Compute/virtualMachines/n1q8y3e1s2j2m9q-np-1-spldz-2nngd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n1q8y3e1s2j2m9q-np-1-spldz-2nngd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-2njpnbrz5jg5--managed/providers/Microsoft.Compute/virtualMachines/n1q8y3e1s2j2m9q-np-1-spldz-2nngd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009bd5e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2njpnbrz5jg5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e034572f-71f9-4484-a06b-08d411aba3d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e034572f-71f9-4484-a06b-08d411aba3d9",
      "name": "e034572f-71f9-4484-a06b-08d411aba3d9",
      "status": "Failed",
      "startTime": "2026-09-29T14:28:57.448756437Z",
      "endTime": "2026-09-29T14:48:03.628370174Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n1q8y3e1s2j2m9q-np-1-spldz-2nngd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-2njpnbrz5jg5--managed/providers/Microsoft.Compute/virtualMachines/n1q8y3e1s2j2m9q-np-1-spldz-2nngd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n1q8y3e1s2j2m9q-np-1-spldz-2nngd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-2njpnbrz5jg5--managed/providers/Microsoft.Compute/virtualMachines/n1q8y3e1s2j2m9q-np-1-spldz-2nngd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 22m 40s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westus3 e2e failed n/a 1
e2e (1)
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
10:13:56 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc00061eb40>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc00061eb40>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
2h 12m 43s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westus3 n/a passed n/a 0 n/a
2h 20m 14s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westus3 n/a passed n/a 0 n/a