CIHealth Run Log

Environment: PROD

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:00:29	1h 54m 59s	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	canadacentral	e2e	failed	n/a	14	e2e (14) ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc000f4b7a0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-5rjr485pnvns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc000f4b7a0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-5rjr485pnvns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc001dfc720>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-72flmh48df4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc001dfc720>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-72flmh48df4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-47sckvjgzdjp/cluster-nightly-4-22-wwdkq6 should provision Unexpected error: <fmt.wrapError \| 0xc000d874a0>: failed to create HCP cluster cluster-nightly-4-22-wwdkq6: failed waiting for cluster="cluster-nightly-4-22-wwdkq6" in resourcegroup="rg-nightly-4-22-47sckvjgzdjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9", "name": "4b93cf23-0f8f-4a86-986d-b583c88b0cd9", "status": "Failed", "startTime": "2026-10-01T23:33:29.768708196Z", "endTime": "2026-10-02T00:02:35.156751805Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-47sckvjgzdjp/cluster-nightly-4-22-wwdkq6 should provision Unexpected error: <fmt.wrapError \| 0xc000d874a0>: failed to create HCP cluster cluster-nightly-4-22-wwdkq6: failed waiting for cluster="cluster-nightly-4-22-wwdkq6" in resourcegroup="rg-nightly-4-22-47sckvjgzdjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9", "name": "4b93cf23-0f8f-4a86-986d-b583c88b0cd9", "status": "Failed", "startTime": "2026-10-01T23:33:29.768708196Z", "endTime": "2026-10-02T00:02:35.156751805Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000b3d180>: failed to create HCP cluster np-dg-minor-7t94dm: failed waiting for cluster="np-dg-minor-7t94dm" in resourcegroup="rg-np-dg-minor-7t94dm-kmpzpbhvdzdx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b", "name": "5b29ed00-0226-42af-9fb9-c6317a83dd8b", "status": "Failed", "startTime": "2026-10-02T00:05:37.505434862Z", "endTime": "2026-10-02T00:34:39.296827861Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000b3d180>: failed to create HCP cluster np-dg-minor-7t94dm: failed waiting for cluster="np-dg-minor-7t94dm" in resourcegroup="rg-np-dg-minor-7t94dm-kmpzpbhvdzdx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b", "name": "5b29ed00-0226-42af-9fb9-c6317a83dd8b", "status": "Failed", "startTime": "2026-10-02T00:05:37.505434862Z", "endTime": "2026-10-02T00:34:39.296827861Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc0013b7a10>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-55xcrpmcpjd5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc0013b7a10>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-55xcrpmcpjd5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-szptjh" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc000f34e40>: failed to create HCP cluster cp-ystream-upgrade-4-23-szptjh: failed waiting for cluster="cp-ystream-upgrade-4-23-szptjh" in resourcegroup="rg-cp-ystream-upgrade-4-23-46pm69d84hvp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c", "name": "780e4e8f-ebe0-42ed-942e-9333a940dc3c", "status": "Failed", "startTime": "2026-10-01T23:37:58.369447145Z", "endTime": "2026-10-02T00:06:59.154886423Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-szptjh" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc000f34e40>: failed to create HCP cluster cp-ystream-upgrade-4-23-szptjh: failed waiting for cluster="cp-ystream-upgrade-4-23-szptjh" in resourcegroup="rg-cp-ystream-upgrade-4-23-46pm69d84hvp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c", "name": "780e4e8f-ebe0-42ed-942e-9333a940dc3c", "status": "Failed", "startTime": "2026-10-01T23:37:58.369447145Z", "endTime": "2026-10-02T00:06:59.154886423Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400 Unexpected error: <fmt.wrapError \| 0xc000ed4500>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ngvxnx" in resourcegroup="rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ngvxnx/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400 Unexpected error: <fmt.wrapError \| 0xc000ed4500>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ngvxnx" in resourcegroup="rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ngvxnx/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" Unexpected error: <fmt.wrapError \| 0xc0012f2520>: failed to create NodePool np-ks75xq: failed starting nodepool creation "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" in resourcegroup="rg-nightly-4-20-2zfvw4gzpm77": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2zfvw4gzpm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-ks75xq/nodePools/np-ks75xq -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" Unexpected error: <fmt.wrapError \| 0xc0012f2520>: failed to create NodePool np-ks75xq: failed starting nodepool creation "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" in resourcegroup="rg-nightly-4-20-2zfvw4gzpm77": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2zfvw4gzpm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-ks75xq/nodePools/np-ks75xq -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc000a6f140>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwrdblk56ldw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc000a6f140>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwrdblk56ldw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-5vfvkg" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc00092f060>: failed to create HCP cluster cp-ystream-upgrade-5-0-5vfvkg: failed waiting for cluster="cp-ystream-upgrade-5-0-5vfvkg" in resourcegroup="rg-cp-ystream-upgrade-5-0-c88htx5v6jhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c", "name": "e8198907-58f9-403b-976d-4cda3e2ce98c", "status": "Failed", "startTime": "2026-10-01T23:38:13.440452696Z", "endTime": "2026-10-02T00:07:19.146440053Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-5vfvkg" at install version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc00092f060>: failed to create HCP cluster cp-ystream-upgrade-5-0-5vfvkg: failed waiting for cluster="cp-ystream-upgrade-5-0-5vfvkg" in resourcegroup="rg-cp-ystream-upgrade-5-0-c88htx5v6jhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c", "name": "e8198907-58f9-403b-976d-4cda3e2ce98c", "status": "Failed", "startTime": "2026-10-01T23:38:13.440452696Z", "endTime": "2026-10-02T00:07:19.146440053Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 23:00:29 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s. Expected success, but got an error: <errors.joinError \| 0xc0014996b0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-10-01-083859, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s. Expected success, but got an error: <errors.joinError \| 0xc0014996b0>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-10-01-083859, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436" ... ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-bjmvhq at version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc0010869e0>: failed to create HCP cluster np-skip-minor-bjmvhq: failed waiting for cluster="np-skip-minor-bjmvhq" in resourcegroup="rg-np-skip-minor-bjmvhq-v7kckbmqz7jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3", "name": "94714044-9af3-4f81-b892-f8e47c0efbb3", "status": "Failed", "startTime": "2026-10-02T00:05:39.688922023Z", "endTime": "2026-10-02T00:34:39.296960872Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-bjmvhq at version 4.22.0-0.nightly-multi-2026-07-08-192436 Unexpected error: <fmt.wrapError \| 0xc0010869e0>: failed to create HCP cluster np-skip-minor-bjmvhq: failed waiting for cluster="np-skip-minor-bjmvhq" in resourcegroup="rg-np-skip-minor-bjmvhq-v7kckbmqz7jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3", "name": "94714044-9af3-4f81-b892-f8e47c0efbb3", "status": "Failed", "startTime": "2026-10-02T00:05:39.688922023Z", "endTime": "2026-10-02T00:34:39.296960872Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:29 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc001138270>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-r822p4wshpfb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc001138270>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-r822p4wshpfb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 23:00:29 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc001797968>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7fp2gpvrmpv5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc", "name": "fbace7f5-c615-4f66-9959-02331d194dcc", "status": "Failed", "startTime": "2026-10-01T23:17:05.438751852Z", "endTime": "2026-10-01T23:36:05.105086237Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster" Unexpected error: <errors.joinError \| 0xc001797968>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7fp2gpvrmpv5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc", "name": "fbace7f5-c615-4f66-9959-02331d194dcc", "status": "Failed", "startTime": "2026-10-01T23:17:05.438751852Z", "endTime": "2026-10-01T23:36:05.105086237Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n" } } -------------------------------------------------------------------------------- ... occurred
15:53:01	2h 49m 51s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	westeurope	n/a	passed	n/a	0	n/a
15:51:39	3h 4m 0s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	eastus2	e2e	failed	n/a	4	e2e (4) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 15:51:39 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6" Unexpected error: <fmt.wrapError \| 0xc0010769c0>: failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g5k9xrk5kbzf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680", "name": "1864d250-37c2-44c1-bbbf-426f923fe680", "status": "Failed", "startTime": "2026-10-01T16:25:59.009326346Z", "endTime": "2026-10-01T16:45:03.114189778Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6" Unexpected error: <fmt.wrapError \| 0xc0010769c0>: failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g5k9xrk5kbzf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680", "name": "1864d250-37c2-44c1-bbbf-426f923fe680", "status": "Failed", "startTime": "2026-10-01T16:25:59.009326346Z", "endTime": "2026-10-01T16:45:03.114189778Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 15:51:39 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000cbe8e0>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-f85th5" in resourcegroup="rg-np-noedge-f85th5-2lj4rn8xdzrh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160", "name": "3cf330e8-6cc6-4d77-bd07-24fe8abc4160", "status": "Failed", "startTime": "2026-10-01T18:02:34.984361176Z", "endTime": "2026-10-01T18:21:38.6870989Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000cbe8e0>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-f85th5" in resourcegroup="rg-np-noedge-f85th5-2lj4rn8xdzrh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160", "name": "3cf330e8-6cc6-4d77-bd07-24fe8abc4160", "status": "Failed", "startTime": "2026-10-01T18:02:34.984361176Z", "endTime": "2026-10-01T18:21:38.6870989Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:51:39 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001802258>: resource group "rg-zstream-upgrade-4-20-26f5nw6rjl84": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-m4xqll" in resourcegroup="rg-zstream-upgrade-4-20-26f5nw6rjl84" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5", "name": "fdb2e512-dda1-4736-993b-cb6fd011fea5", "status": "Failed", "startTime": "2026-10-01T17:28:43.347525555Z", "endTime": "2026-10-01T17:52:46.138429989Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t648pvv6ett9l7cgmh574bcs7i8oe4q still exists (deletion dispatched at 2026-10-01T17:28:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001802258>: resource group "rg-zstream-upgrade-4-20-26f5nw6rjl84": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-m4xqll" in resourcegroup="rg-zstream-upgrade-4-20-26f5nw6rjl84" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5", "name": "fdb2e512-dda1-4736-993b-cb6fd011fea5", "status": "Failed", "startTime": "2026-10-01T17:28:43.347525555Z", "endTime": "2026-10-01T17:52:46.138429989Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t648pvv6ett9l7cgmh574bcs7i8oe4q still exists (deletion dispatched at 2026-10-01T17:28:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect 15:51:39 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region Unexpected error: <fmt.wrapError \| 0xc001f80160>: selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region Unexpected error: <fmt.wrapError \| 0xc001f80160>: selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect ... occurred
07:37:13	2h 29m 19s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	brazilsouth	n/a	passed	n/a	0	n/a
04:49:27	2h 43m 15s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	westeurope	e2e	failed	n/a	1	e2e (1) ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift 04:49:27 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster" Unexpected error: <fmt.wrapError \| 0xc000638600>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2nzmgkxwzltg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535", "name": "eaa1d5d5-fc6a-4816-a207-694d27453535", "status": "Failed", "startTime": "2026-10-01T05:17:35.937388036Z", "endTime": "2026-10-01T05:36:39.648087048Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: Creating: virtualmachine creating or updating\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster" Unexpected error: <fmt.wrapError \| 0xc000638600>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2nzmgkxwzltg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535", "name": "eaa1d5d5-fc6a-4816-a207-694d27453535", "status": "Failed", "startTime": "2026-10-01T05:17:35.937388036Z", "endTime": "2026-10-01T05:36:39.648087048Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: Creating: virtualmachine creating or updating\n" } } -------------------------------------------------------------------------------- ... occurred
04:47:40	2h 44m 26s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	brazilsouth	e2e	failed	n/a	1	e2e (1) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 04:47:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw" Unexpected error: <fmt.wrapError \| 0xc0005cc920>: failed to create NodePool np-ttzgdw: failed waiting for nodepool="np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw" in resourcegroup="rg-candidate-4-23-2gh8rvt7nqts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e", "name": "77801f28-f3c8-4c43-a265-b1074104615e", "status": "Failed", "startTime": "2026-10-01T05:58:38.023066435Z", "endTime": "2026-10-01T06:17:40.99775088Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw" Unexpected error: <fmt.wrapError \| 0xc0005cc920>: failed to create NodePool np-ttzgdw: failed waiting for nodepool="np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw" in resourcegroup="rg-candidate-4-23-2gh8rvt7nqts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e", "name": "77801f28-f3c8-4c43-a265-b1074104615e", "status": "Failed", "startTime": "2026-10-01T05:58:38.023066435Z", "endTime": "2026-10-01T06:17:40.99775088Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n" } } -------------------------------------------------------------------------------- ... occurred
04:40:13	2h 57m 42s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	eastus2	e2e	failed	n/a	3	e2e (3) ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 04:40:13 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc000f94940>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-shmswfdhjbs5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f", "name": "5bf51a3f-7e39-4185-a2e3-46167169303f", "status": "Failed", "startTime": "2026-10-01T05:34:42.041881998Z", "endTime": "2026-10-01T05:53:44.016651744Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc000f94940>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-shmswfdhjbs5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f", "name": "5bf51a3f-7e39-4185-a2e3-46167169303f", "status": "Failed", "startTime": "2026-10-01T05:34:42.041881998Z", "endTime": "2026-10-01T05:53:44.016651744Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n" } } -------------------------------------------------------------------------------- ... occurred selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect 04:40:13 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region Unexpected error: <fmt.wrapError \| 0xc0012cb640>: selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region Unexpected error: <fmt.wrapError \| 0xc0012cb640>: selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect ... occurred ERROR CODE: DeadlineExceeded; provider Microsoft.Compute 04:40:13 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc002d00360>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kssqgrfmgk4t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c", "name": "fef1a167-cbd5-4042-994e-c159a5e7bb3c", "status": "Failed", "startTime": "2026-10-01T05:33:21.995938468Z", "endTime": "2026-10-01T05:52:24.037057581Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc002d00360>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kssqgrfmgk4t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c", "name": "fef1a167-cbd5-4042-994e-c159a5e7bb3c", "status": "Failed", "startTime": "2026-10-01T05:33:21.995938468Z", "endTime": "2026-10-01T05:52:24.037057581Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n" } } -------------------------------------------------------------------------------- ... occurred
02:00:28	2h 15m 24s	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	australiaeast	e2e	failed	n/a	10	e2e (10) ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-kbkkg2" to 4.21 Unexpected error: <exported.ResponseError \| 0xc00140fc80>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-fw66sfpzcwgg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-kbkkg2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-kbkkg2" to 4.21 Unexpected error: <exported.ResponseError \| 0xc00140fc80>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-fw66sfpzcwgg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-kbkkg2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xwvg9x" to follow minor version 5.0 Unexpected error: <exported.ResponseError \| 0xc000e80e40>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-n4rqqc24skhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xwvg9x -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xwvg9x" to follow minor version 5.0 Unexpected error: <exported.ResponseError \| 0xc000e80e40>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-n4rqqc24skhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xwvg9x -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7ktnhw" to 4.22 Unexpected error: <exported.ResponseError \| 0xc00104ef60>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-qqs544jdcqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7ktnhw -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7ktnhw" to 4.22 Unexpected error: <exported.ResponseError \| 0xc00104ef60>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-qqs544jdcqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7ktnhw -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-24sw5t" to 4.23 Unexpected error: <exported.ResponseError \| 0xc000cdd5c0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vhz5sh8l55g9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-24sw5t -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-24sw5t" to 4.23 Unexpected error: <exported.ResponseError \| 0xc000cdd5c0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vhz5sh8l55g9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-24sw5t -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-7wwtdz" to follow minor version 4.22 Unexpected error: <exported.ResponseError \| 0xc0012d2210>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-w7lfjqpj8rr7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-7wwtdz -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-7wwtdz" to follow minor version 4.22 Unexpected error: <exported.ResponseError \| 0xc0012d2210>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-w7lfjqpj8rr7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-7wwtdz -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-gdj89p" to follow minor version 4.20 Unexpected error: <exported.ResponseError \| 0xc00157dbf0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rb86w8ww8sjw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gdj89p -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-gdj89p" to follow minor version 4.20 Unexpected error: <exported.ResponseError \| 0xc00157dbf0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rb86w8ww8sjw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gdj89p -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4mwqz2" to follow minor version 4.23 Unexpected error: <exported.ResponseError \| 0xc00144a120>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jlsd79zkk6hh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4mwqz2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4mwqz2" to follow minor version 4.23 Unexpected error: <exported.ResponseError \| 0xc00144a120>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jlsd79zkk6hh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4mwqz2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-l9fj7m" to 5.0 Unexpected error: <exported.ResponseError \| 0xc0015e41b0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-4mvbdm9qd7qf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-l9fj7m -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-l9fj7m" to 5.0 Unexpected error: <exported.ResponseError \| 0xc0015e41b0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-4mvbdm9qd7qf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-l9fj7m -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-4fwf7q" to follow minor version 4.21 Unexpected error: <exported.ResponseError \| 0xc001226630>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wnp2jmpdpwln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4fwf7q -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-4fwf7q" to follow minor version 4.21 Unexpected error: <exported.ResponseError \| 0xc001226630>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wnp2jmpdpwln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4fwf7q -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift 02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-gvphp6" to 5.1 Unexpected error: <exported.ResponseError \| 0xc0010f8390>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qgd4bkmfx2lq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-gvphp6 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-gvphp6" to 5.1 Unexpected error: <exported.ResponseError \| 0xc0010f8390>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qgd4bkmfx2lq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-gvphp6 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidNullTagValue -------------------------------------------------------------------------------- { "error": { "code": "InvalidNullTagValue", "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null." } } -------------------------------------------------------------------------------- ... occurred
01:00:54	2h 38m 22s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	centralindia	n/a	passed	n/a	0	n/a
00:55:44	2h 47m 56s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	switzerlandnorth	n/a	passed	n/a	0	n/a
00:53:35	2h 50m 2s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	australiaeast	n/a	passed	n/a	0	n/a
00:47:09	2h 41m 52s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	canadacentral	n/a	passed	n/a	0	n/a