CIHealth Run Log

Environment: STG

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:00:17	1h 59m 6s	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	17	e2e (17) ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-vg7tn9" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc000f98ae0>: failed to create HCP cluster cp-ystream-upgrade-5-0-vg7tn9: failed waiting for cluster="cp-ystream-upgrade-5-0-vg7tn9" in resourcegroup="rg-cp-ystream-upgrade-5-0-rxgcfxnprf4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dfb91f8-fcd3-478b-b03f-ad43d9671cdb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dfb91f8-fcd3-478b-b03f-ad43d9671cdb", "name": "4dfb91f8-fcd3-478b-b03f-ad43d9671cdb", "status": "Failed", "startTime": "2026-09-26T23:49:36.517184321Z", "endTime": "2026-09-27T00:18:40.854855035Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-vg7tn9" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc000f98ae0>: failed to create HCP cluster cp-ystream-upgrade-5-0-vg7tn9: failed waiting for cluster="cp-ystream-upgrade-5-0-vg7tn9" in resourcegroup="rg-cp-ystream-upgrade-5-0-rxgcfxnprf4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dfb91f8-fcd3-478b-b03f-ad43d9671cdb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dfb91f8-fcd3-478b-b03f-ad43d9671cdb", "name": "4dfb91f8-fcd3-478b-b03f-ad43d9671cdb", "status": "Failed", "startTime": "2026-09-26T23:49:36.517184321Z", "endTime": "2026-09-27T00:18:40.854855035Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc001c13890>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-xslh5ql6dz6k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc001c13890>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-xslh5ql6dz6k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc001207440>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-p68sqdmczpdg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc001207440>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-p68sqdmczpdg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000afa620>: failed to create HCP cluster np-dg-minor-6vv4rp: failed waiting for cluster="np-dg-minor-6vv4rp" in resourcegroup="rg-np-dg-minor-6vv4rp-259t47rttggq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66c0cba9-056f-4636-8679-f28ce98cdca9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66c0cba9-056f-4636-8679-f28ce98cdca9", "name": "66c0cba9-056f-4636-8679-f28ce98cdca9", "status": "Failed", "startTime": "2026-09-27T00:05:12.968666607Z", "endTime": "2026-09-27T00:34:17.912617084Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000afa620>: failed to create HCP cluster np-dg-minor-6vv4rp: failed waiting for cluster="np-dg-minor-6vv4rp" in resourcegroup="rg-np-dg-minor-6vv4rp-259t47rttggq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66c0cba9-056f-4636-8679-f28ce98cdca9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66c0cba9-056f-4636-8679-f28ce98cdca9", "name": "66c0cba9-056f-4636-8679-f28ce98cdca9", "status": "Failed", "startTime": "2026-09-27T00:05:12.968666607Z", "endTime": "2026-09-27T00:34:17.912617084Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-f6zhb5 at version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc00011b080>: failed to create HCP cluster np-skip-minor-f6zhb5: failed waiting for cluster="np-skip-minor-f6zhb5" in resourcegroup="rg-np-skip-minor-f6zhb5-mltd4bpn2ng8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3d18d82-d0f9-496a-97bb-ff0fdefcdec6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3d18d82-d0f9-496a-97bb-ff0fdefcdec6", "name": "b3d18d82-d0f9-496a-97bb-ff0fdefcdec6", "status": "Failed", "startTime": "2026-09-27T00:03:26.334155746Z", "endTime": "2026-09-27T00:32:27.905092762Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-f6zhb5 at version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc00011b080>: failed to create HCP cluster np-skip-minor-f6zhb5: failed waiting for cluster="np-skip-minor-f6zhb5" in resourcegroup="rg-np-skip-minor-f6zhb5-mltd4bpn2ng8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3d18d82-d0f9-496a-97bb-ff0fdefcdec6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3d18d82-d0f9-496a-97bb-ff0fdefcdec6", "name": "b3d18d82-d0f9-496a-97bb-ff0fdefcdec6", "status": "Failed", "startTime": "2026-09-27T00:03:26.334155746Z", "endTime": "2026-09-27T00:32:27.905092762Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-v9gzmf" for cluster "cluster-nightly-4-20-v9gzmf" Unexpected error: <fmt.wrapError \| 0xc0016ad720>: failed to create NodePool np-v9gzmf: failed starting nodepool creation "np-v9gzmf" for cluster "cluster-nightly-4-20-v9gzmf" in resourcegroup="rg-nightly-4-20-lq47jh9j7pkp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-lq47jh9j7pkp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-v9gzmf/nodePools/np-v9gzmf -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-v9gzmf" for cluster "cluster-nightly-4-20-v9gzmf" Unexpected error: <fmt.wrapError \| 0xc0016ad720>: failed to create NodePool np-v9gzmf: failed starting nodepool creation "np-v9gzmf" for cluster "cluster-nightly-4-20-v9gzmf" in resourcegroup="rg-nightly-4-20-lq47jh9j7pkp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-lq47jh9j7pkp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-v9gzmf/nodePools/np-v9gzmf -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc001b9f6e0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-5phhxmkf5cjv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc001b9f6e0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-5phhxmkf5cjv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc0005e2bc0>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-czbz85" in resourcegroup="rg-np-version-upgrade-czbz85-h7z4wz569lk2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-czbz85-h7z4wz569lk2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-czbz85/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc0005e2bc0>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-czbz85" in resourcegroup="rg-np-version-upgrade-czbz85-h7z4wz569lk2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-czbz85-h7z4wz569lk2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-czbz85/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc001849320>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7h9j6j7d894p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc001849320>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7h9j6j7d894p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 23:00:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s. Expected success, but got an error: <errors.joinError \| 0xc00141cf78>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-09-24-033657, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s. Expected success, but got an error: <errors.joinError \| 0xc00141cf78>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-09-24-033657, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436" ... ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc0014f6150>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-zmbfdjmggw6v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc0014f6150>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-zmbfdjmggw6v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-tdqnbhv945ln/cluster-nightly-4-22-hwtgkf should provision Unexpected error: <fmt.wrapError \| 0xc000936960>: failed to create HCP cluster cluster-nightly-4-22-hwtgkf: failed waiting for cluster="cluster-nightly-4-22-hwtgkf" in resourcegroup="rg-nightly-4-22-tdqnbhv945ln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c8f5770-a2ce-4214-855c-9d70572faf14 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c8f5770-a2ce-4214-855c-9d70572faf14", "name": "2c8f5770-a2ce-4214-855c-9d70572faf14", "status": "Failed", "startTime": "2026-09-26T23:43:51.36895754Z", "endTime": "2026-09-27T00:13:00.824555826Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-tdqnbhv945ln/cluster-nightly-4-22-hwtgkf should provision Unexpected error: <fmt.wrapError \| 0xc000936960>: failed to create HCP cluster cluster-nightly-4-22-hwtgkf: failed waiting for cluster="cluster-nightly-4-22-hwtgkf" in resourcegroup="rg-nightly-4-22-tdqnbhv945ln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c8f5770-a2ce-4214-855c-9d70572faf14 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c8f5770-a2ce-4214-855c-9d70572faf14", "name": "2c8f5770-a2ce-4214-855c-9d70572faf14", "status": "Failed", "startTime": "2026-09-26T23:43:51.36895754Z", "endTime": "2026-09-27T00:13:00.824555826Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 23:00:17 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc00011a380>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-kb4s8cpxl5xw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66b0c680-744e-463d-83fe-32a6b03b0059 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66b0c680-744e-463d-83fe-32a6b03b0059", "name": "66b0c680-744e-463d-83fe-32a6b03b0059", "status": "Failed", "startTime": "2026-09-26T23:30:24.096246355Z", "endTime": "2026-09-26T23:49:30.686897638Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m1h7k2a6w9l9r1k-np-1-hqxhh-kj929: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-kb4s8cpxl5xw--managed/providers/Microsoft.Compute/virtualMachines/m1h7k2a6w9l9r1k-np-1-hqxhh-kj929 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m1h7k2a6w9l9r1k-np-1-hqxhh-kj929: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-kb4s8cpxl5xw--managed/providers/Microsoft.Compute/virtualMachines/m1h7k2a6w9l9r1k-np-1-hqxhh-kj929 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc00011a380>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-kb4s8cpxl5xw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66b0c680-744e-463d-83fe-32a6b03b0059 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66b0c680-744e-463d-83fe-32a6b03b0059", "name": "66b0c680-744e-463d-83fe-32a6b03b0059", "status": "Failed", "startTime": "2026-09-26T23:30:24.096246355Z", "endTime": "2026-09-26T23:49:30.686897638Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m1h7k2a6w9l9r1k-np-1-hqxhh-kj929: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-kb4s8cpxl5xw--managed/providers/Microsoft.Compute/virtualMachines/m1h7k2a6w9l9r1k-np-1-hqxhh-kj929 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m1h7k2a6w9l9r1k-np-1-hqxhh-kj929: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-kb4s8cpxl5xw--managed/providers/Microsoft.Compute/virtualMachines/m1h7k2a6w9l9r1k-np-1-hqxhh-kj929 to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-8f7dzbcrbw7p/cluster-nightly-5-1-b2l6kn should provision Unexpected error: <fmt.wrapError \| 0xc000f37180>: failed to create HCP cluster cluster-nightly-5-1-b2l6kn: failed waiting for cluster="cluster-nightly-5-1-b2l6kn" in resourcegroup="rg-nightly-5-1-8f7dzbcrbw7p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e80d0504-5f18-4937-9363-cfe3fda28970 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e80d0504-5f18-4937-9363-cfe3fda28970", "name": "e80d0504-5f18-4937-9363-cfe3fda28970", "status": "Failed", "startTime": "2026-09-26T23:46:02.695503366Z", "endTime": "2026-09-27T00:15:10.822999259Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-8f7dzbcrbw7p/cluster-nightly-5-1-b2l6kn should provision Unexpected error: <fmt.wrapError \| 0xc000f37180>: failed to create HCP cluster cluster-nightly-5-1-b2l6kn: failed waiting for cluster="cluster-nightly-5-1-b2l6kn" in resourcegroup="rg-nightly-5-1-8f7dzbcrbw7p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e80d0504-5f18-4937-9363-cfe3fda28970 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e80d0504-5f18-4937-9363-cfe3fda28970", "name": "e80d0504-5f18-4937-9363-cfe3fda28970", "status": "Failed", "startTime": "2026-09-26T23:46:02.695503366Z", "endTime": "2026-09-27T00:15:10.822999259Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 23:00:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.004s. Expected success, but got an error: <errors.joinError \| 0xc0017240c0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.004s. Expected success, but got an error: <errors.joinError \| 0xc0017240c0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053" ... ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 23:00:17 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc001907188>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hpx4nnz688cq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5c2ff73-ec69-46ee-a311-273a07e5930d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5c2ff73-ec69-46ee-a311-273a07e5930d", "name": "d5c2ff73-ec69-46ee-a311-273a07e5930d", "status": "Failed", "startTime": "2026-09-26T23:33:38.153702381Z", "endTime": "2026-09-26T23:52:40.648818601Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cdrhx-29bn6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hpx4nnz688cq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cdrhx-29bn6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cdrhx-29bn6: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hpx4nnz688cq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cdrhx-29bn6 to exist\n" } } -------------------------------------------------------------------------------- encountered errors downloading console logs: - VM "cilium-cluster-cilium-np-cdrhx-29bn6": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-cdrhx-29bn6" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hpx4nnz688cq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cdrhx-29bn6/retrieveBootDiagnosticsData -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: OperationNotAllowed -------------------------------------------------------------------------------- { "error": { "code": "OperationNotAllowed", "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-cdrhx-29bn6' since another operation is in progress." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc001907188>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hpx4nnz688cq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5c2ff73-ec69-46ee-a311-273a07e5930d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5c2ff73-ec69-46ee-a311-273a07e5930d", "name": "d5c2ff73-ec69-46ee-a311-273a07e5930d", "status": "Failed", "startTime": "2026-09-26T23:33:38.153702381Z", "endTime": "2026-09-26T23:52:40.648818601Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cdrhx-29bn6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hpx4nnz688cq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cdrhx-29bn6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cdrhx-29bn6: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hpx4nnz688cq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cdrhx-29bn6 to exist\n" } } -------------------------------------------------------------------------------- encountered errors downloading console logs: - VM "cilium-cluster-cilium-np-cdrhx-29bn6": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-cdrhx-29bn6" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hpx4nnz688cq--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cdrhx-29bn6/retrieveBootDiagnosticsData -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: OperationNotAllowed -------------------------------------------------------------------------------- { "error": { "code": "OperationNotAllowed", "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-cdrhx-29bn6' since another operation is in progress." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bk2bfx" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc00089aa80>: failed to create HCP cluster cp-ystream-upgrade-4-23-bk2bfx: failed waiting for cluster="cp-ystream-upgrade-4-23-bk2bfx" in resourcegroup="rg-cp-ystream-upgrade-4-23-t8gzs6dzd69p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c52163c-38c9-4f22-9154-361c3d2553ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c52163c-38c9-4f22-9154-361c3d2553ba", "name": "9c52163c-38c9-4f22-9154-361c3d2553ba", "status": "Failed", "startTime": "2026-09-26T23:49:13.237430377Z", "endTime": "2026-09-27T00:18:20.853759656Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bk2bfx" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc00089aa80>: failed to create HCP cluster cp-ystream-upgrade-4-23-bk2bfx: failed waiting for cluster="cp-ystream-upgrade-4-23-bk2bfx" in resourcegroup="rg-cp-ystream-upgrade-4-23-t8gzs6dzd69p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c52163c-38c9-4f22-9154-361c3d2553ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c52163c-38c9-4f22-9154-361c3d2553ba", "name": "9c52163c-38c9-4f22-9154-361c3d2553ba", "status": "Failed", "startTime": "2026-09-26T23:49:13.237430377Z", "endTime": "2026-09-27T00:18:20.853759656Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred
21:14:06	2h 4m 51s	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	3	e2e (3) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 21:14:06 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc0028b2360>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-brtk2h8rsjng" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc36dfc4-12f6-4dff-948b-9d2f00f4cdab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc36dfc4-12f6-4dff-948b-9d2f00f4cdab", "name": "bc36dfc4-12f6-4dff-948b-9d2f00f4cdab", "status": "Failed", "startTime": "2026-09-26T21:29:04.025452385Z", "endTime": "2026-09-26T21:48:08.955970797Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-sxr66-dqnpn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-brtk2h8rsjng--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sxr66-dqnpn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-sxr66-dqnpn: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-brtk2h8rsjng--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sxr66-dqnpn to exist\n" } } -------------------------------------------------------------------------------- encountered errors downloading console logs: - VM "cilium-cluster-cilium-np-sxr66-dqnpn": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-sxr66-dqnpn" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-brtk2h8rsjng--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sxr66-dqnpn/retrieveBootDiagnosticsData -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: OperationNotAllowed -------------------------------------------------------------------------------- { "error": { "code": "OperationNotAllowed", "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-sxr66-dqnpn' since another operation is in progress." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc0028b2360>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-brtk2h8rsjng" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc36dfc4-12f6-4dff-948b-9d2f00f4cdab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc36dfc4-12f6-4dff-948b-9d2f00f4cdab", "name": "bc36dfc4-12f6-4dff-948b-9d2f00f4cdab", "status": "Failed", "startTime": "2026-09-26T21:29:04.025452385Z", "endTime": "2026-09-26T21:48:08.955970797Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-sxr66-dqnpn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-brtk2h8rsjng--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sxr66-dqnpn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-sxr66-dqnpn: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-brtk2h8rsjng--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sxr66-dqnpn to exist\n" } } -------------------------------------------------------------------------------- encountered errors downloading console logs: - VM "cilium-cluster-cilium-np-sxr66-dqnpn": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-sxr66-dqnpn" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-brtk2h8rsjng--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sxr66-dqnpn/retrieveBootDiagnosticsData -------------------------------------------------------------------------------- RESPONSE 409: 409 Conflict ERROR CODE: OperationNotAllowed -------------------------------------------------------------------------------- { "error": { "code": "OperationNotAllowed", "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-sxr66-dqnpn' since another operation is in progress." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 21:14:06 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:598]: failed to create nodepool npdg-4-21 Unexpected error: <fmt.wrapError \| 0xc000915360>: failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-94wr55" in resourcegroup="rg-np-downgrade-94wr55-49w8dwsv7jfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/65b789c4-2968-40e8-b87b-ea9ebda5d678 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/65b789c4-2968-40e8-b87b-ea9ebda5d678", "name": "65b789c4-2968-40e8-b87b-ea9ebda5d678", "status": "Failed", "startTime": "2026-09-26T22:23:09.106743803Z", "endTime": "2026-09-26T22:42:12.893995625Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-94wr55-49w8dwsv7jfz-np-dg-94wr55--managed/providers/Microsoft.Compute/virtualMachines/s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-94wr55-49w8dwsv7jfz-np-dg-94wr55--managed/providers/Microsoft.Compute/virtualMachines/s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:598]: failed to create nodepool npdg-4-21 Unexpected error: <fmt.wrapError \| 0xc000915360>: failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-94wr55" in resourcegroup="rg-np-downgrade-94wr55-49w8dwsv7jfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/65b789c4-2968-40e8-b87b-ea9ebda5d678 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/65b789c4-2968-40e8-b87b-ea9ebda5d678", "name": "65b789c4-2968-40e8-b87b-ea9ebda5d678", "status": "Failed", "startTime": "2026-09-26T22:23:09.106743803Z", "endTime": "2026-09-26T22:42:12.893995625Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-94wr55-49w8dwsv7jfz-np-dg-94wr55--managed/providers/Microsoft.Compute/virtualMachines/s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-94wr55-49w8dwsv7jfz-np-dg-94wr55--managed/providers/Microsoft.Compute/virtualMachines/s5n9q0l4z0q6e9r-npdg-4-21-k5pss-hmspx to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 21:14:06 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0014c6168>: resource group "tls-endpoint-cluster-s8s7nfpcrjx5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-s8s7nfpcrjx5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f", "name": "add1f9fb-6a8e-4417-8717-b28b0f76087f", "status": "Failed", "startTime": "2026-09-26T21:34:10.468425235Z", "endTime": "2026-09-26T21:58:19.111858781Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2ugn3l1bvjlm39njkp5ap0fh8ek88r still exists (deletion dispatched at 2026-09-26T21:34:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0014c6168>: resource group "tls-endpoint-cluster-s8s7nfpcrjx5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-s8s7nfpcrjx5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f", "name": "add1f9fb-6a8e-4417-8717-b28b0f76087f", "status": "Failed", "startTime": "2026-09-26T21:34:10.468425235Z", "endTime": "2026-09-26T21:58:19.111858781Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2ugn3l1bvjlm39njkp5ap0fh8ek88r still exists (deletion dispatched at 2026-09-26T21:34:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- ... occurred
19:02:17	2h 11m 14s	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	2	e2e (2) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 19:02:17 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1 Unexpected error: <fmt.wrapError \| 0xc001417120>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-hz29dh4r44cz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5d69a1b-6093-4eff-815f-ec78ccfb28e7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5d69a1b-6093-4eff-815f-ec78ccfb28e7", "name": "d5d69a1b-6093-4eff-815f-ec78ccfb28e7", "status": "Failed", "startTime": "2026-09-26T20:29:03.064278124Z", "endTime": "2026-09-26T20:48:03.684881036Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-hz29dh4r44cz--managed/providers/Microsoft.Compute/virtualMachines/y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-hz29dh4r44cz--managed/providers/Microsoft.Compute/virtualMachines/y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1 Unexpected error: <fmt.wrapError \| 0xc001417120>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-hz29dh4r44cz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5d69a1b-6093-4eff-815f-ec78ccfb28e7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5d69a1b-6093-4eff-815f-ec78ccfb28e7", "name": "d5d69a1b-6093-4eff-815f-ec78ccfb28e7", "status": "Failed", "startTime": "2026-09-26T20:29:03.064278124Z", "endTime": "2026-09-26T20:48:03.684881036Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-hz29dh4r44cz--managed/providers/Microsoft.Compute/virtualMachines/y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-hz29dh4r44cz--managed/providers/Microsoft.Compute/virtualMachines/y2z0t7s5v3i5o8p-np-1-pm27w-q5ghp to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 19:02:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk Unexpected error: <fmt.wrapError \| 0xc00101c7e0>: failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-wqmzcwndlf2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/287407fa-d792-43c8-a3ac-bf8d179a2257 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/287407fa-d792-43c8-a3ac-bf8d179a2257", "name": "287407fa-d792-43c8-a3ac-bf8d179a2257", "status": "Failed", "startTime": "2026-09-26T19:30:11.133018346Z", "endTime": "2026-09-26T19:49:20.223545331Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-wqmzcwndlf2p--managed/providers/Microsoft.Compute/virtualMachines/z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-wqmzcwndlf2p--managed/providers/Microsoft.Compute/virtualMachines/z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk Unexpected error: <fmt.wrapError \| 0xc00101c7e0>: failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-wqmzcwndlf2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/287407fa-d792-43c8-a3ac-bf8d179a2257 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/287407fa-d792-43c8-a3ac-bf8d179a2257", "name": "287407fa-d792-43c8-a3ac-bf8d179a2257", "status": "Failed", "startTime": "2026-09-26T19:30:11.133018346Z", "endTime": "2026-09-26T19:49:20.223545331Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-wqmzcwndlf2p--managed/providers/Microsoft.Compute/virtualMachines/z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-wqmzcwndlf2p--managed/providers/Microsoft.Compute/virtualMachines/z7b9z5h4y0c0o9n-nodepool-128gib-7dvff-gjh67 to exist\n" } } -------------------------------------------------------------------------------- ... occurred
14:41:27	2h 12m 24s	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	6	e2e (6) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 14:41:27 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc00107c880>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-xnlxm6q7lcm9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ff23c508-1534-45b0-afe1-322eade0805c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ff23c508-1534-45b0-afe1-322eade0805c", "name": "ff23c508-1534-45b0-afe1-322eade0805c", "status": "Failed", "startTime": "2026-09-26T15:14:28.73622764Z", "endTime": "2026-09-26T15:33:28.526950876Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-xnlxm6q7lcm9--managed/providers/Microsoft.Compute/virtualMachines/t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-xnlxm6q7lcm9--managed/providers/Microsoft.Compute/virtualMachines/t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc00107c880>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-xnlxm6q7lcm9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ff23c508-1534-45b0-afe1-322eade0805c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ff23c508-1534-45b0-afe1-322eade0805c", "name": "ff23c508-1534-45b0-afe1-322eade0805c", "status": "Failed", "startTime": "2026-09-26T15:14:28.73622764Z", "endTime": "2026-09-26T15:33:28.526950876Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-xnlxm6q7lcm9--managed/providers/Microsoft.Compute/virtualMachines/t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-xnlxm6q7lcm9--managed/providers/Microsoft.Compute/virtualMachines/t3g2u3v4h9d5k0f-np-1-kr8jm-h74pm to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 14:41:27 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl" Unexpected error: <errors.joinError \| 0xc001af6f48>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-xwhrsdh6tmlv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f6fb1ff-b199-44fc-b474-a371b575c211 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f6fb1ff-b199-44fc-b474-a371b575c211", "name": "7f6fb1ff-b199-44fc-b474-a371b575c211", "status": "Failed", "startTime": "2026-09-26T15:13:41.359494364Z", "endTime": "2026-09-26T15:32:48.541665354Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-mhn9k-npbsd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-xwhrsdh6tmlv--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-mhn9k-npbsd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-mhn9k-npbsd: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-xwhrsdh6tmlv--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-mhn9k-npbsd to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl" Unexpected error: <errors.joinError \| 0xc001af6f48>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-xwhrsdh6tmlv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f6fb1ff-b199-44fc-b474-a371b575c211 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f6fb1ff-b199-44fc-b474-a371b575c211", "name": "7f6fb1ff-b199-44fc-b474-a371b575c211", "status": "Failed", "startTime": "2026-09-26T15:13:41.359494364Z", "endTime": "2026-09-26T15:32:48.541665354Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-mhn9k-npbsd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-xwhrsdh6tmlv--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-mhn9k-npbsd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-mhn9k-npbsd: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-xwhrsdh6tmlv--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-mhn9k-npbsd to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 14:41:27 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000a52a80>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-z8x96q" in resourcegroup="rg-np-noedge-z8x96q-2kdtqk7hg4n9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9feb1206-42e5-45be-9e7f-937293863c77 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9feb1206-42e5-45be-9e7f-937293863c77", "name": "9feb1206-42e5-45be-9e7f-937293863c77", "status": "Failed", "startTime": "2026-09-26T16:09:21.39874787Z", "endTime": "2026-09-26T16:28:25.629531487Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-z8x96q-2kdtqk7hg4n9-np-ne-z8x96q--managed/providers/Microsoft.Compute/virtualMachines/z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-z8x96q-2kdtqk7hg4n9-np-ne-z8x96q--managed/providers/Microsoft.Compute/virtualMachines/z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000a52a80>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-z8x96q" in resourcegroup="rg-np-noedge-z8x96q-2kdtqk7hg4n9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9feb1206-42e5-45be-9e7f-937293863c77 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9feb1206-42e5-45be-9e7f-937293863c77", "name": "9feb1206-42e5-45be-9e7f-937293863c77", "status": "Failed", "startTime": "2026-09-26T16:09:21.39874787Z", "endTime": "2026-09-26T16:28:25.629531487Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-z8x96q-2kdtqk7hg4n9-np-ne-z8x96q--managed/providers/Microsoft.Compute/virtualMachines/z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-z8x96q-2kdtqk7hg4n9-np-ne-z8x96q--managed/providers/Microsoft.Compute/virtualMachines/z6z5b4r3d3c5x2u-npnoedge-4-20-hkdwv-g8gpq to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 14:41:27 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.40 Unexpected error: <fmt.wrapError \| 0xc000d436e0>: failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-v7prvk" in resourcegroup="rg-np-skip-minor-v7prvk-5b2q5bj8c6pf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85e9d6dd-362f-4b59-9640-d13f74c43941 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85e9d6dd-362f-4b59-9640-d13f74c43941", "name": "85e9d6dd-362f-4b59-9640-d13f74c43941", "status": "Failed", "startTime": "2026-09-26T16:07:25.346522094Z", "endTime": "2026-09-26T16:26:25.857169591Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-v7prvk-5b2q5bj8c6pf-np-sm-v7prvk--managed/providers/Microsoft.Compute/virtualMachines/i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-v7prvk-5b2q5bj8c6pf-np-sm-v7prvk--managed/providers/Microsoft.Compute/virtualMachines/i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.40 Unexpected error: <fmt.wrapError \| 0xc000d436e0>: failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-v7prvk" in resourcegroup="rg-np-skip-minor-v7prvk-5b2q5bj8c6pf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85e9d6dd-362f-4b59-9640-d13f74c43941 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85e9d6dd-362f-4b59-9640-d13f74c43941", "name": "85e9d6dd-362f-4b59-9640-d13f74c43941", "status": "Failed", "startTime": "2026-09-26T16:07:25.346522094Z", "endTime": "2026-09-26T16:26:25.857169591Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-v7prvk-5b2q5bj8c6pf-np-sm-v7prvk--managed/providers/Microsoft.Compute/virtualMachines/i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-v7prvk-5b2q5bj8c6pf-np-sm-v7prvk--managed/providers/Microsoft.Compute/virtualMachines/i4x5r0s3e6z4u1b-nps-420-422-r55vg-fj7nb to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 14:41:27 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling Unexpected error: <fmt.wrapError \| 0xc000535720>: failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mtj55995hz55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ae86850-245b-4016-bb60-b4b035485db0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ae86850-245b-4016-bb60-b4b035485db0", "name": "8ae86850-245b-4016-bb60-b4b035485db0", "status": "Failed", "startTime": "2026-09-26T15:57:07.959371171Z", "endTime": "2026-09-26T16:16:15.682428941Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-mtj55995hz55--managed/providers/Microsoft.Compute/virtualMachines/i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-mtj55995hz55--managed/providers/Microsoft.Compute/virtualMachines/i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling Unexpected error: <fmt.wrapError \| 0xc000535720>: failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mtj55995hz55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ae86850-245b-4016-bb60-b4b035485db0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ae86850-245b-4016-bb60-b4b035485db0", "name": "8ae86850-245b-4016-bb60-b4b035485db0", "status": "Failed", "startTime": "2026-09-26T15:57:07.959371171Z", "endTime": "2026-09-26T16:16:15.682428941Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-mtj55995hz55--managed/providers/Microsoft.Compute/virtualMachines/i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-mtj55995hz55--managed/providers/Microsoft.Compute/virtualMachines/i5e8l8z4l0r7p7c-autoscale-noaz-mdlgl-vsxt8 to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 14:41:27 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6" Unexpected error: <fmt.wrapError \| 0xc000b44360>: failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-j2mzj6vms9k6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1c9994c-76dc-4ba9-a74c-2e9f6ee89b81 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1c9994c-76dc-4ba9-a74c-2e9f6ee89b81", "name": "e1c9994c-76dc-4ba9-a74c-2e9f6ee89b81", "status": "Failed", "startTime": "2026-09-26T15:13:01.589549739Z", "endTime": "2026-09-26T15:32:08.767838782Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-j2mzj6vms9k6--managed/providers/Microsoft.Compute/virtualMachines/t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-j2mzj6vms9k6--managed/providers/Microsoft.Compute/virtualMachines/t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6" Unexpected error: <fmt.wrapError \| 0xc000b44360>: failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-j2mzj6vms9k6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1c9994c-76dc-4ba9-a74c-2e9f6ee89b81 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1c9994c-76dc-4ba9-a74c-2e9f6ee89b81", "name": "e1c9994c-76dc-4ba9-a74c-2e9f6ee89b81", "status": "Failed", "startTime": "2026-09-26T15:13:01.589549739Z", "endTime": "2026-09-26T15:32:08.767838782Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-j2mzj6vms9k6--managed/providers/Microsoft.Compute/virtualMachines/t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-j2mzj6vms9k6--managed/providers/Microsoft.Compute/virtualMachines/t6o2m0h2e0w4n7z-arm64-vm-np-1-kfrdk-p9nsm to exist\n" } } -------------------------------------------------------------------------------- ... occurred
09:28:15	4h 37m 21s	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	3	e2e (3) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 09:28:15 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1 Unexpected error: <fmt.wrapError \| 0xc0001199c0>: failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-rzctzfm7wj8d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d62a8c5-db27-49e9-941f-20e83e0e72a4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d62a8c5-db27-49e9-941f-20e83e0e72a4", "name": "6d62a8c5-db27-49e9-941f-20e83e0e72a4", "status": "Failed", "startTime": "2026-09-26T10:32:28.860402325Z", "endTime": "2026-09-26T10:51:36.244471948Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-hl4rt-rwrt6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-rzctzfm7wj8d--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-hl4rt-rwrt6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-hl4rt-rwrt6: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-rzctzfm7wj8d--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-hl4rt-rwrt6 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1 Unexpected error: <fmt.wrapError \| 0xc0001199c0>: failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-rzctzfm7wj8d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d62a8c5-db27-49e9-941f-20e83e0e72a4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d62a8c5-db27-49e9-941f-20e83e0e72a4", "name": "6d62a8c5-db27-49e9-941f-20e83e0e72a4", "status": "Failed", "startTime": "2026-09-26T10:32:28.860402325Z", "endTime": "2026-09-26T10:51:36.244471948Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-hl4rt-rwrt6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-rzctzfm7wj8d--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-hl4rt-rwrt6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-hl4rt-rwrt6: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-rzctzfm7wj8d--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-hl4rt-rwrt6 to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 09:28:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a00138>: resource group "pullsecret-test-5vdpdhbfbqdk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5vdpdhbfbqdk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4", "name": "82ed94ae-6b5c-42a4-9f90-3280fa2efff4", "status": "Failed", "startTime": "2026-09-26T10:04:15.110849507Z", "endTime": "2026-09-26T10:28:16.325232898Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdg1r6crhcm349aur60m1j1fcbuh2 still exists (deletion dispatched at 2026-09-26T10:04:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a00138>: resource group "pullsecret-test-5vdpdhbfbqdk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5vdpdhbfbqdk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4", "name": "82ed94ae-6b5c-42a4-9f90-3280fa2efff4", "status": "Failed", "startTime": "2026-09-26T10:04:15.110849507Z", "endTime": "2026-09-26T10:28:16.325232898Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdg1r6crhcm349aur60m1j1fcbuh2 still exists (deletion dispatched at 2026-09-26T10:04:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 09:28:15 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001c140c0>: resource group "tls-endpoint-cluster-tlzh4bzgw4zh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-tlzh4bzgw4zh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8", "name": "3547c17b-2baf-4dfa-a1b2-187f3ed372c8", "status": "Failed", "startTime": "2026-09-26T10:04:17.573277188Z", "endTime": "2026-09-26T10:28:26.278499332Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdfqqgc9osqgq67u99r2mbueuta1t still exists (deletion dispatched at 2026-09-26T10:04:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001c140c0>: resource group "tls-endpoint-cluster-tlzh4bzgw4zh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-tlzh4bzgw4zh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8", "name": "3547c17b-2baf-4dfa-a1b2-187f3ed372c8", "status": "Failed", "startTime": "2026-09-26T10:04:17.573277188Z", "endTime": "2026-09-26T10:28:26.278499332Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdfqqgc9osqgq67u99r2mbueuta1t still exists (deletion dispatched at 2026-09-26T10:04:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
05:55:28	1h 53m 14s	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	2	e2e (2) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 05:55:28 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tcwpsk" for cluster "cluster-candidate-4-21-tcwpsk" Unexpected error: <fmt.wrapError \| 0xc000219700>: failed to create NodePool np-tcwpsk: failed waiting for nodepool="np-tcwpsk" for cluster "cluster-candidate-4-21-tcwpsk" in resourcegroup="rg-candidate-4-21-fcmqz6t9cprm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6626f1a6-57db-468a-9a1e-3a38d2373d87 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6626f1a6-57db-468a-9a1e-3a38d2373d87", "name": "6626f1a6-57db-468a-9a1e-3a38d2373d87", "status": "Failed", "startTime": "2026-09-26T06:35:50.80138522Z", "endTime": "2026-09-26T06:54:54.507616611Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-fcmqz6t9cprm-candidate-tcwpsk--managed/providers/Microsoft.Compute/virtualMachines/u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-fcmqz6t9cprm-candidate-tcwpsk--managed/providers/Microsoft.Compute/virtualMachines/u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tcwpsk" for cluster "cluster-candidate-4-21-tcwpsk" Unexpected error: <fmt.wrapError \| 0xc000219700>: failed to create NodePool np-tcwpsk: failed waiting for nodepool="np-tcwpsk" for cluster "cluster-candidate-4-21-tcwpsk" in resourcegroup="rg-candidate-4-21-fcmqz6t9cprm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6626f1a6-57db-468a-9a1e-3a38d2373d87 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6626f1a6-57db-468a-9a1e-3a38d2373d87", "name": "6626f1a6-57db-468a-9a1e-3a38d2373d87", "status": "Failed", "startTime": "2026-09-26T06:35:50.80138522Z", "endTime": "2026-09-26T06:54:54.507616611Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-fcmqz6t9cprm-candidate-tcwpsk--managed/providers/Microsoft.Compute/virtualMachines/u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-fcmqz6t9cprm-candidate-tcwpsk--managed/providers/Microsoft.Compute/virtualMachines/u4m5g1d1o4y5j2t-np-tcwpsk-474vr-hcjmx to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 05:55:28 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc000b1aea0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-l5wcbthtdvrj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc4520f0-59dd-4c27-b6ac-3a37c3d68b61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc4520f0-59dd-4c27-b6ac-3a37c3d68b61", "name": "fc4520f0-59dd-4c27-b6ac-3a37c3d68b61", "status": "Failed", "startTime": "2026-09-26T06:10:39.440155769Z", "endTime": "2026-09-26T06:29:48.524896029Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-l5wcbthtdvrj--managed/providers/Microsoft.Compute/virtualMachines/r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-l5wcbthtdvrj--managed/providers/Microsoft.Compute/virtualMachines/r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster" Unexpected error: <fmt.wrapError \| 0xc000b1aea0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-l5wcbthtdvrj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc4520f0-59dd-4c27-b6ac-3a37c3d68b61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc4520f0-59dd-4c27-b6ac-3a37c3d68b61", "name": "fc4520f0-59dd-4c27-b6ac-3a37c3d68b61", "status": "Failed", "startTime": "2026-09-26T06:10:39.440155769Z", "endTime": "2026-09-26T06:29:48.524896029Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-l5wcbthtdvrj--managed/providers/Microsoft.Compute/virtualMachines/r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-l5wcbthtdvrj--managed/providers/Microsoft.Compute/virtualMachines/r2a4d5d1z0i7s2r-np-1-9q5wc-dlv85 to exist\n" } } -------------------------------------------------------------------------------- ... occurred
03:36:04	2h 13m 27s	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	1	e2e (1) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 03:36:04 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2c4h96" for cluster "cluster-candidate-4-23-2c4h96" Unexpected error: <fmt.wrapError \| 0xc000a68b20>: failed to create NodePool np-2c4h96: failed waiting for nodepool="np-2c4h96" for cluster "cluster-candidate-4-23-2c4h96" in resourcegroup="rg-candidate-4-23-54tf9r44kbdx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/084dd236-2858-409a-b7ec-4a1409b0e6b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/084dd236-2858-409a-b7ec-4a1409b0e6b6", "name": "084dd236-2858-409a-b7ec-4a1409b0e6b6", "status": "Failed", "startTime": "2026-09-26T04:37:04.007379792Z", "endTime": "2026-09-26T04:56:04.017294795Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-54tf9r44kbdx-candidate-2c4h96--managed/providers/Microsoft.Compute/virtualMachines/h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-54tf9r44kbdx-candidate-2c4h96--managed/providers/Microsoft.Compute/virtualMachines/h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2c4h96" for cluster "cluster-candidate-4-23-2c4h96" Unexpected error: <fmt.wrapError \| 0xc000a68b20>: failed to create NodePool np-2c4h96: failed waiting for nodepool="np-2c4h96" for cluster "cluster-candidate-4-23-2c4h96" in resourcegroup="rg-candidate-4-23-54tf9r44kbdx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/084dd236-2858-409a-b7ec-4a1409b0e6b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/084dd236-2858-409a-b7ec-4a1409b0e6b6", "name": "084dd236-2858-409a-b7ec-4a1409b0e6b6", "status": "Failed", "startTime": "2026-09-26T04:37:04.007379792Z", "endTime": "2026-09-26T04:56:04.017294795Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-54tf9r44kbdx-candidate-2c4h96--managed/providers/Microsoft.Compute/virtualMachines/h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-54tf9r44kbdx-candidate-2c4h96--managed/providers/Microsoft.Compute/virtualMachines/h4g8t4x4z5g9w3y-np-2c4h96-btvvm-wk24r to exist\n" } } -------------------------------------------------------------------------------- ... occurred
02:00:16	1h 53m 51s	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	2	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute 02:00:16 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000b56dc0>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-c2cwvz" in resourcegroup="rg-np-noedge-c2cwvz-qj8wgrzd8jpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca", "name": "4d8efa06-245b-47cb-9abb-ec98064be1ca", "status": "Failed", "startTime": "2026-09-26T03:07:12.674137276Z", "endTime": "2026-09-26T03:26:19.484267097Z", "error": { "code": "InternalServerError", "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000b56dc0>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-c2cwvz" in resourcegroup="rg-np-noedge-c2cwvz-qj8wgrzd8jpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca", "name": "4d8efa06-245b-47cb-9abb-ec98064be1ca", "status": "Failed", "startTime": "2026-09-26T03:07:12.674137276Z", "endTime": "2026-09-26T03:26:19.484267097Z", "error": { "code": "InternalServerError", "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 02:00:16 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np Unexpected error: <fmt.wrapError \| 0xc000c1ba60>: failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-89w5k67x22pw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d1d982d-3b36-4d74-abe3-4a35c064fbed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d1d982d-3b36-4d74-abe3-4a35c064fbed", "name": "6d1d982d-3b36-4d74-abe3-4a35c064fbed", "status": "Failed", "startTime": "2026-09-26T03:09:42.046607083Z", "endTime": "2026-09-26T03:28:46.487681317Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-89w5k67x22pw--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-89w5k67x22pw--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np Unexpected error: <fmt.wrapError \| 0xc000c1ba60>: failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-89w5k67x22pw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d1d982d-3b36-4d74-abe3-4a35c064fbed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d1d982d-3b36-4d74-abe3-4a35c064fbed", "name": "6d1d982d-3b36-4d74-abe3-4a35c064fbed", "status": "Failed", "startTime": "2026-09-26T03:09:42.046607083Z", "endTime": "2026-09-26T03:28:46.487681317Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-89w5k67x22pw--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-89w5k67x22pw--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-jb75r-zxj2z to exist\n" } } -------------------------------------------------------------------------------- ... occurred