CIHealth Run Log

Environment: PROD

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:00:27	1h 50m 0s	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	canadacentral	e2e	failed	n/a	13	e2e (13) ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc0025a1e90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g9r75fj7b84l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc0025a1e90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g9r75fj7b84l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 23:00:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000ec1320>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cbk98wj7cx2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91", "name": "a41762b8-220b-4e53-b3be-168dfb531e91", "status": "Failed", "startTime": "2026-09-29T23:10:58.443702839Z", "endTime": "2026-09-29T23:30:01.681918109Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000ec1320>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cbk98wj7cx2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91", "name": "a41762b8-220b-4e53-b3be-168dfb531e91", "status": "Failed", "startTime": "2026-09-29T23:10:58.443702839Z", "endTime": "2026-09-29T23:30:01.681918109Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n" } } -------------------------------------------------------------------------------- ... occurred VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 23:00:27 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s. Expected success, but got an error: <errors.joinError \| 0xc0016fb248>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-110018, targetMinor=5.1.0-0.nightly-multi-2026-09-29-141926) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-29-141926" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s. Expected success, but got an error: <errors.joinError \| 0xc0016fb248>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-110018, targetMinor=5.1.0-0.nightly-multi-2026-09-29-141926) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-29-141926" ... ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-76zbc6" for cluster "cluster-nightly-4-20-76zbc6" Unexpected error: <fmt.wrapError \| 0xc0010d40e0>: failed to create NodePool np-76zbc6: failed starting nodepool creation "np-76zbc6" for cluster "cluster-nightly-4-20-76zbc6" in resourcegroup="rg-nightly-4-20-tmbd62wjjmv5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-tmbd62wjjmv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-76zbc6/nodePools/np-76zbc6 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-76zbc6" for cluster "cluster-nightly-4-20-76zbc6" Unexpected error: <fmt.wrapError \| 0xc0010d40e0>: failed to create NodePool np-76zbc6: failed starting nodepool creation "np-76zbc6" for cluster "cluster-nightly-4-20-76zbc6" in resourcegroup="rg-nightly-4-20-tmbd62wjjmv5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-tmbd62wjjmv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-76zbc6/nodePools/np-76zbc6 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc0012dbc40>: failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-hz8wft" resourcegroup="rg-np-dg-minor-hz8wft-2td2rb8jcw6z": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hz8wft-2td2rb8jcw6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hz8wft/nodePools/npdg-422-420 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc0012dbc40>: failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-hz8wft" resourcegroup="rg-np-dg-minor-hz8wft-2td2rb8jcw6z": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hz8wft-2td2rb8jcw6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hz8wft/nodePools/npdg-422-420 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc002153e30>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-ndd4ps9rz69q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc002153e30>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-ndd4ps9rz69q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> 23:00:27 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools Expected success, but got an error: <fmt.wrapError \| 0xc000d87400>: failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cg4srqjqkk9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2", "name": "49815947-d474-46af-9a2a-1b7e7d2030b2", "status": "Failed", "startTime": "2026-09-29T23:56:01.394801153Z", "endTime": "2026-09-30T00:15:03.380581382Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n" } } -------------------------------------------------------------------------------- ... fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools Expected success, but got an error: <fmt.wrapError \| 0xc000d87400>: failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cg4srqjqkk9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2", "name": "49815947-d474-46af-9a2a-1b7e7d2030b2", "status": "Failed", "startTime": "2026-09-29T23:56:01.394801153Z", "endTime": "2026-09-30T00:15:03.380581382Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n" } } -------------------------------------------------------------------------------- ... VerifyBasicAccess failed: failed to list services: Unauthorized 23:00:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-21-lhp5d5" is viable Unexpected error: <errors.joinError \| 0xc001347770>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized VerifyBasicAccess failed: failed to list services: Unauthorized ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-21-lhp5d5" is viable Unexpected error: <errors.joinError \| 0xc001347770>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized VerifyBasicAccess failed: failed to list services: Unauthorized ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000800580>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-v4bb8q" in resourcegroup="rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-v4bb8q/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000800580>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-v4bb8q" in resourcegroup="rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-v4bb8q/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc0029fc4b0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-bgcgl9s2h6qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc0029fc4b0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-bgcgl9s2h6qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000fb6040>: failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-dzh29w" in resourcegroup="rg-np-skip-minor-dzh29w-lnc5s2hc2m99": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dzh29w-lnc5s2hc2m99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dzh29w/nodePools/nps-420-422 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000fb6040>: failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-dzh29w" in resourcegroup="rg-np-skip-minor-dzh29w-lnc5s2hc2m99": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dzh29w-lnc5s2hc2m99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dzh29w/nodePools/nps-420-422 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc0015ed110>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-scbxjjvdnhlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc0015ed110>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-scbxjjvdnhlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc000fcf3e0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-mnsnt8965m6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc000fcf3e0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-mnsnt8965m6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred
16:04:17	2h 31m 21s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	4	e2e (4) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 16:04:17 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420 Unexpected error: <fmt.wrapError \| 0xc001022120>: failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-kvm5ll" in resourcegroup="rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44", "name": "42494991-e3af-48e6-9cf3-a4fc4d279c44", "status": "Failed", "startTime": "2026-09-29T17:48:38.482509384Z", "endTime": "2026-09-29T18:07:42.157683685Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420 Unexpected error: <fmt.wrapError \| 0xc001022120>: failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-kvm5ll" in resourcegroup="rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44", "name": "42494991-e3af-48e6-9cf3-a4fc4d279c44", "status": "Failed", "startTime": "2026-09-29T17:48:38.482509384Z", "endTime": "2026-09-29T18:07:42.157683685Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 16:04:17 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress" Unexpected error: <fmt.wrapError \| 0xc000f3a060>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-fld425vx7n2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5", "name": "84d85252-8c39-447d-8d5d-a497997a37f5", "status": "Failed", "startTime": "2026-09-29T16:34:49.001481538Z", "endTime": "2026-09-29T16:53:58.101757577Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-l7x62-h42h7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-l7x62-h42h7: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress" Unexpected error: <fmt.wrapError \| 0xc000f3a060>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-fld425vx7n2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5", "name": "84d85252-8c39-447d-8d5d-a497997a37f5", "status": "Failed", "startTime": "2026-09-29T16:34:49.001481538Z", "endTime": "2026-09-29T16:53:58.101757577Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-l7x62-h42h7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-l7x62-h42h7: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n" } } -------------------------------------------------------------------------------- ... occurred failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> 16:04:17 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools Expected success, but got an error: <fmt.wrapError \| 0xc00079ab80>: failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lwltp7hhv9xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9", "name": "ade802f3-fc4a-4348-8b7b-a05cecc41cb9", "status": "Failed", "startTime": "2026-09-29T17:40:59.55523927Z", "endTime": "2026-09-29T18:00:02.471778007Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n" } } -------------------------------------------------------------------------------- ... fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools Expected success, but got an error: <fmt.wrapError \| 0xc00079ab80>: failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lwltp7hhv9xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9", "name": "ade802f3-fc4a-4348-8b7b-a05cecc41cb9", "status": "Failed", "startTime": "2026-09-29T17:40:59.55523927Z", "endTime": "2026-09-29T18:00:02.471778007Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n" } } -------------------------------------------------------------------------------- ... VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 16:04:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s. Expected success, but got an error: <errors.joinError \| 0xc000deacc0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s. Expected success, but got an error: <errors.joinError \| 0xc000deacc0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1" ...
12:23:47	2h 36m 17s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	5	e2e (5) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 12:23:47 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000f52a00>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-42jhzb" in resourcegroup="rg-np-noedge-42jhzb-sgqdpkmgvtp6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b11a0a62-0163-43ee-bbd8-7683a7459b17 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b11a0a62-0163-43ee-bbd8-7683a7459b17", "name": "b11a0a62-0163-43ee-bbd8-7683a7459b17", "status": "Failed", "startTime": "2026-09-29T14:07:28.714225296Z", "endTime": "2026-09-29T14:26:31.083712333Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-42jhzb-sgqdpkmgvtp6-np-ne-42jhzb--managed/providers/Microsoft.Compute/virtualMachines/z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-42jhzb-sgqdpkmgvtp6-np-ne-42jhzb--managed/providers/Microsoft.Compute/virtualMachines/z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000f52a00>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-42jhzb" in resourcegroup="rg-np-noedge-42jhzb-sgqdpkmgvtp6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b11a0a62-0163-43ee-bbd8-7683a7459b17 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b11a0a62-0163-43ee-bbd8-7683a7459b17", "name": "b11a0a62-0163-43ee-bbd8-7683a7459b17", "status": "Failed", "startTime": "2026-09-29T14:07:28.714225296Z", "endTime": "2026-09-29T14:26:31.083712333Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-42jhzb-sgqdpkmgvtp6-np-ne-42jhzb--managed/providers/Microsoft.Compute/virtualMachines/z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-42jhzb-sgqdpkmgvtp6-np-ne-42jhzb--managed/providers/Microsoft.Compute/virtualMachines/z6e0a9g3g7e7q4w-npnoedge-4-20-jjt24-gstcv to exist\n" } } -------------------------------------------------------------------------------- ... occurred failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> 12:23:47 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools Expected success, but got an error: <fmt.wrapError \| 0xc000dfcbc0>: failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r2grpqzmpxnk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93", "name": "e2b31ee5-5378-49f4-b4f9-8c82f9bace93", "status": "Failed", "startTime": "2026-09-29T13:59:14.617140907Z", "endTime": "2026-09-29T14:18:21.033986186Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n" } } -------------------------------------------------------------------------------- ... fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools Expected success, but got an error: <fmt.wrapError \| 0xc000dfcbc0>: failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r2grpqzmpxnk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93", "name": "e2b31ee5-5378-49f4-b4f9-8c82f9bace93", "status": "Failed", "startTime": "2026-09-29T13:59:14.617140907Z", "endTime": "2026-09-29T14:18:21.033986186Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n" } } -------------------------------------------------------------------------------- ... VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 12:23:47 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s. Expected success, but got an error: <errors.joinError \| 0xc0014784b0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s. Expected success, but got an error: <errors.joinError \| 0xc0014784b0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1" ... ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 12:23:47 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster Unexpected error: <fmt.wrapError \| 0xc000c09f00>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qds642pj6qdc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8c7a098f-68ef-4837-bf5c-33fe66729db0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8c7a098f-68ef-4837-bf5c-33fe66729db0", "name": "8c7a098f-68ef-4837-bf5c-33fe66729db0", "status": "Failed", "startTime": "2026-09-29T12:58:44.310890224Z", "endTime": "2026-09-29T13:17:52.410585343Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x4e8d7k0u2h1t8g-np-1-4stht-tcn7q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qds642pj6qdc--managed/providers/Microsoft.Compute/virtualMachines/x4e8d7k0u2h1t8g-np-1-4stht-tcn7q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x4e8d7k0u2h1t8g-np-1-4stht-tcn7q: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qds642pj6qdc--managed/providers/Microsoft.Compute/virtualMachines/x4e8d7k0u2h1t8g-np-1-4stht-tcn7q to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster Unexpected error: <fmt.wrapError \| 0xc000c09f00>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qds642pj6qdc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8c7a098f-68ef-4837-bf5c-33fe66729db0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8c7a098f-68ef-4837-bf5c-33fe66729db0", "name": "8c7a098f-68ef-4837-bf5c-33fe66729db0", "status": "Failed", "startTime": "2026-09-29T12:58:44.310890224Z", "endTime": "2026-09-29T13:17:52.410585343Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x4e8d7k0u2h1t8g-np-1-4stht-tcn7q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qds642pj6qdc--managed/providers/Microsoft.Compute/virtualMachines/x4e8d7k0u2h1t8g-np-1-4stht-tcn7q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x4e8d7k0u2h1t8g-np-1-4stht-tcn7q: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qds642pj6qdc--managed/providers/Microsoft.Compute/virtualMachines/x4e8d7k0u2h1t8g-np-1-4stht-tcn7q to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 12:23:47 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6lgs9c" for cluster "cluster-candidate-4-21-6lgs9c" Unexpected error: <fmt.wrapError \| 0xc000aab760>: failed to create NodePool np-6lgs9c: failed waiting for nodepool="np-6lgs9c" for cluster "cluster-candidate-4-21-6lgs9c" in resourcegroup="rg-candidate-4-21-ptntl59g9fpg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c0c5da21-b394-4076-91ad-570f20be81a7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c0c5da21-b394-4076-91ad-570f20be81a7", "name": "c0c5da21-b394-4076-91ad-570f20be81a7", "status": "Failed", "startTime": "2026-09-29T13:23:06.981213639Z", "endTime": "2026-09-29T13:42:09.219679905Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ptntl59g9fpg-candidate-6lgs9c--managed/providers/Microsoft.Compute/virtualMachines/p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ptntl59g9fpg-candidate-6lgs9c--managed/providers/Microsoft.Compute/virtualMachines/p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6lgs9c" for cluster "cluster-candidate-4-21-6lgs9c" Unexpected error: <fmt.wrapError \| 0xc000aab760>: failed to create NodePool np-6lgs9c: failed waiting for nodepool="np-6lgs9c" for cluster "cluster-candidate-4-21-6lgs9c" in resourcegroup="rg-candidate-4-21-ptntl59g9fpg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c0c5da21-b394-4076-91ad-570f20be81a7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c0c5da21-b394-4076-91ad-570f20be81a7", "name": "c0c5da21-b394-4076-91ad-570f20be81a7", "status": "Failed", "startTime": "2026-09-29T13:23:06.981213639Z", "endTime": "2026-09-29T13:42:09.219679905Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ptntl59g9fpg-candidate-6lgs9c--managed/providers/Microsoft.Compute/virtualMachines/p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ptntl59g9fpg-candidate-6lgs9c--managed/providers/Microsoft.Compute/virtualMachines/p3t9k3i6s5x8k4x-np-6lgs9c-jwnkm-t8k7p to exist\n" } } -------------------------------------------------------------------------------- ... occurred
06:27:12	2h 35m 9s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	eastus2	e2e	failed	n/a	2	e2e (2) selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect 06:27:12 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region Unexpected error: <fmt.wrapError \| 0xc001352fa0>: selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region Unexpected error: <fmt.wrapError \| 0xc001352fa0>: selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect ... occurred GET <url> 06:27:12 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:258]: failed to install cert-manager manifests Expected success, but got an error: <errors.errorString \| 0xc000b64160>: GET https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml: HTTP 500 ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:258]: failed to install cert-manager manifests Expected success, but got an error: <errors.errorString \| 0xc000b64160>: GET https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml: HTTP 500 ...
02:46:19	2h 26m 37s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	westeurope	n/a	passed	n/a	0	n/a
02:00:26	2h 8m 6s	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	3	e2e (3) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 02:00:26 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fktv4d" for cluster "cluster-candidate-4-21-fktv4d" Unexpected error: <fmt.wrapError \| 0xc0010608e0>: failed to create NodePool np-fktv4d: failed waiting for nodepool="np-fktv4d" for cluster "cluster-candidate-4-21-fktv4d" in resourcegroup="rg-candidate-4-21-hd26bdgtn7gq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01d13c95-d010-4354-af72-1e3f0c8961b4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01d13c95-d010-4354-af72-1e3f0c8961b4", "name": "01d13c95-d010-4354-af72-1e3f0c8961b4", "status": "Failed", "startTime": "2026-09-29T02:40:40.344373047Z", "endTime": "2026-09-29T02:59:45.460904082Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hd26bdgtn7gq-candidate-fktv4d--managed/providers/Microsoft.Compute/virtualMachines/e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hd26bdgtn7gq-candidate-fktv4d--managed/providers/Microsoft.Compute/virtualMachines/e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fktv4d" for cluster "cluster-candidate-4-21-fktv4d" Unexpected error: <fmt.wrapError \| 0xc0010608e0>: failed to create NodePool np-fktv4d: failed waiting for nodepool="np-fktv4d" for cluster "cluster-candidate-4-21-fktv4d" in resourcegroup="rg-candidate-4-21-hd26bdgtn7gq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01d13c95-d010-4354-af72-1e3f0c8961b4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01d13c95-d010-4354-af72-1e3f0c8961b4", "name": "01d13c95-d010-4354-af72-1e3f0c8961b4", "status": "Failed", "startTime": "2026-09-29T02:40:40.344373047Z", "endTime": "2026-09-29T02:59:45.460904082Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hd26bdgtn7gq-candidate-fktv4d--managed/providers/Microsoft.Compute/virtualMachines/e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hd26bdgtn7gq-candidate-fktv4d--managed/providers/Microsoft.Compute/virtualMachines/e7o6n1m5a6l5q6f-np-fktv4d-k4bkc-pvr2z to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 02:00:26 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1 Unexpected error: <fmt.wrapError \| 0xc00011bb80>: failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-br9bvcrqkdrx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/267eabaf-73d5-414d-b422-0156f94fa98b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/267eabaf-73d5-414d-b422-0156f94fa98b", "name": "267eabaf-73d5-414d-b422-0156f94fa98b", "status": "Failed", "startTime": "2026-09-29T03:00:06.808479674Z", "endTime": "2026-09-29T03:19:12.619464984Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-9q5vb-nv244: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-br9bvcrqkdrx--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-9q5vb-nv244 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-9q5vb-nv244: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-br9bvcrqkdrx--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-9q5vb-nv244 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1 Unexpected error: <fmt.wrapError \| 0xc00011bb80>: failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-br9bvcrqkdrx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/267eabaf-73d5-414d-b422-0156f94fa98b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/267eabaf-73d5-414d-b422-0156f94fa98b", "name": "267eabaf-73d5-414d-b422-0156f94fa98b", "status": "Failed", "startTime": "2026-09-29T03:00:06.808479674Z", "endTime": "2026-09-29T03:19:12.619464984Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-9q5vb-nv244: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-br9bvcrqkdrx--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-9q5vb-nv244 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-9q5vb-nv244: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-br9bvcrqkdrx--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-9q5vb-nv244 to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 02:00:26 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc00083c440>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-j2w7rvh22rc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d195bba-0850-461e-a119-0cb340343650 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d195bba-0850-461e-a119-0cb340343650", "name": "9d195bba-0850-461e-a119-0cb340343650", "status": "Failed", "startTime": "2026-09-29T02:16:42.180371111Z", "endTime": "2026-09-29T02:35:43.192706201Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s5l6m9s6s7e2g7d-np-1-sktsn-kwql2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-j2w7rvh22rc8--managed/providers/Microsoft.Compute/virtualMachines/s5l6m9s6s7e2g7d-np-1-sktsn-kwql2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s5l6m9s6s7e2g7d-np-1-sktsn-kwql2: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-j2w7rvh22rc8--managed/providers/Microsoft.Compute/virtualMachines/s5l6m9s6s7e2g7d-np-1-sktsn-kwql2 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc00083c440>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-j2w7rvh22rc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d195bba-0850-461e-a119-0cb340343650 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d195bba-0850-461e-a119-0cb340343650", "name": "9d195bba-0850-461e-a119-0cb340343650", "status": "Failed", "startTime": "2026-09-29T02:16:42.180371111Z", "endTime": "2026-09-29T02:35:43.192706201Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s5l6m9s6s7e2g7d-np-1-sktsn-kwql2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-j2w7rvh22rc8--managed/providers/Microsoft.Compute/virtualMachines/s5l6m9s6s7e2g7d-np-1-sktsn-kwql2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s5l6m9s6s7e2g7d-np-1-sktsn-kwql2: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-j2w7rvh22rc8--managed/providers/Microsoft.Compute/virtualMachines/s5l6m9s6s7e2g7d-np-1-sktsn-kwql2 to exist\n" } } -------------------------------------------------------------------------------- ... occurred
01:52:10	2h 14m 41s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	eastus2	e2e	failed	n/a	1	e2e (1) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 01:52:10 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk Unexpected error: <fmt.wrapError \| 0xc001005800>: failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-w2rs2wktq5hp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/78d682cc-8178-44fc-8481-457b338bfc70 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/78d682cc-8178-44fc-8481-457b338bfc70", "name": "78d682cc-8178-44fc-8481-457b338bfc70", "status": "Failed", "startTime": "2026-09-29T02:05:25.77800873Z", "endTime": "2026-09-29T02:24:26.035091452Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w2rs2wktq5hp--managed/providers/Microsoft.Compute/virtualMachines/a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w2rs2wktq5hp--managed/providers/Microsoft.Compute/virtualMachines/a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk Unexpected error: <fmt.wrapError \| 0xc001005800>: failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-w2rs2wktq5hp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/78d682cc-8178-44fc-8481-457b338bfc70 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/78d682cc-8178-44fc-8481-457b338bfc70", "name": "78d682cc-8178-44fc-8481-457b338bfc70", "status": "Failed", "startTime": "2026-09-29T02:05:25.77800873Z", "endTime": "2026-09-29T02:24:26.035091452Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w2rs2wktq5hp--managed/providers/Microsoft.Compute/virtualMachines/a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w2rs2wktq5hp--managed/providers/Microsoft.Compute/virtualMachines/a0a1s6r0g7y9m9a-nodepool-128gib-bkb4m-hkxx5 to exist\n" } } -------------------------------------------------------------------------------- ... occurred