CIHealth Run Log

↗ Open Failure patterns for this day

Environment: STG

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:00:48	1h 47m 24s	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	6	e2e (6) ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:48 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0x272ecda2fb90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster correlation ID: 3b21cb42-ff17-49b5-9ed9-e3d69922540a client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0x272ecda2fb90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster correlation ID: 3b21cb42-ff17-49b5-9ed9-e3d69922540a client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:48 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0x3bf246e81b90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster correlation ID: c883dc6d-a815-4804-970f-295c17dcee98 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0x3bf246e81b90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster correlation ID: c883dc6d-a815-4804-970f-295c17dcee98 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:48 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0x25636a4bb950>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster correlation ID: eb7e7c6f-c7e9-46a0-8d7d-c420ec91eb1d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0x25636a4bb950>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster correlation ID: eb7e7c6f-c7e9-46a0-8d7d-c420ec91eb1d client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift 23:00:48 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-kxcsmnps9zfz/cluster-nightly-4-21-j9wst5 should provision Unexpected error: <fmt.wrapError \| 0x21388a648d60>: failed to create HCP cluster cluster-nightly-4-21-j9wst5: failed waiting for cluster="cluster-nightly-4-21-j9wst5" in resourcegroup="rg-nightly-4-21-kxcsmnps9zfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d", "name": "6943378a-a2ac-4799-849e-25f8e97d936d", "status": "Failed", "startTime": "2026-10-13T23:24:54.16266447Z", "endTime": "2026-10-13T23:54:01.266968226Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d correlation ID: 11b5d2e6-bd60-46f6-9cea-eb83bb792173 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-kxcsmnps9zfz/cluster-nightly-4-21-j9wst5 should provision Unexpected error: <fmt.wrapError \| 0x21388a648d60>: failed to create HCP cluster cluster-nightly-4-21-j9wst5: failed waiting for cluster="cluster-nightly-4-21-j9wst5" in resourcegroup="rg-nightly-4-21-kxcsmnps9zfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d", "name": "6943378a-a2ac-4799-849e-25f8e97d936d", "status": "Failed", "startTime": "2026-10-13T23:24:54.16266447Z", "endTime": "2026-10-13T23:54:01.266968226Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d correlation ID: 11b5d2e6-bd60-46f6-9cea-eb83bb792173 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:48 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0x1fbd9ed3dd40>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster correlation ID: 3c34a173-f084-4c74-8bf8-8d8da2ea7e20 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0x1fbd9ed3dd40>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster correlation ID: 3c34a173-f084-4c74-8bf8-8d8da2ea7e20 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:48 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0x1ca0706e2750>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test correlation ID: dd69bc64-32de-4cf3-a00f-c421bb664b95 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0x1ca0706e2750>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test correlation ID: dd69bc64-32de-4cf3-a00f-c421bb664b95 client request ID: unavailable occurred
00:11:54	7h 10m 8s	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	westus3	e2e	failed	n/a	33	e2e (33) ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-dwlbs8wqmf4n/cluster-candidate-5-1-dhmlhx should provision Unexpected error: <fmt.wrapError \| 0x32444a40800>: failed to create HCP cluster cluster-candidate-5-1-dhmlhx: failed waiting for cluster="cluster-candidate-5-1-dhmlhx" in resourcegroup="rg-candidate-5-1-dwlbs8wqmf4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8", "name": "cca958a7-cdd6-4407-a240-933a8c20efd8", "status": "Failed", "startTime": "2026-10-13T00:45:11.941637821Z", "endTime": "2026-10-13T01:14:17.337870809Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8 correlation ID: 4827029f-0d94-4b20-83b7-9db0cfabf792 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-dwlbs8wqmf4n/cluster-candidate-5-1-dhmlhx should provision Unexpected error: <fmt.wrapError \| 0x32444a40800>: failed to create HCP cluster cluster-candidate-5-1-dhmlhx: failed waiting for cluster="cluster-candidate-5-1-dhmlhx" in resourcegroup="rg-candidate-5-1-dwlbs8wqmf4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8", "name": "cca958a7-cdd6-4407-a240-933a8c20efd8", "status": "Failed", "startTime": "2026-10-13T00:45:11.941637821Z", "endTime": "2026-10-13T01:14:17.337870809Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8 correlation ID: 4827029f-0d94-4b20-83b7-9db0cfabf792 client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-24hzxh9hvdbr/cluster-candidate-4-22-dg2r6h should provision Unexpected error: <fmt.wrapError \| 0x1893baafcc80>: failed to create HCP cluster cluster-candidate-4-22-dg2r6h: failed waiting for cluster="cluster-candidate-4-22-dg2r6h" in resourcegroup="rg-candidate-4-22-24hzxh9hvdbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460", "name": "7a5573b4-6d8c-4cfa-88d0-da0d2af71460", "status": "Failed", "startTime": "2026-10-13T00:45:09.767142622Z", "endTime": "2026-10-13T01:14:17.351146196Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460 correlation ID: 60fbb9ec-6da1-4cca-9803-0dd26c08e96f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-24hzxh9hvdbr/cluster-candidate-4-22-dg2r6h should provision Unexpected error: <fmt.wrapError \| 0x1893baafcc80>: failed to create HCP cluster cluster-candidate-4-22-dg2r6h: failed waiting for cluster="cluster-candidate-4-22-dg2r6h" in resourcegroup="rg-candidate-4-22-24hzxh9hvdbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460", "name": "7a5573b4-6d8c-4cfa-88d0-da0d2af71460", "status": "Failed", "startTime": "2026-10-13T00:45:09.767142622Z", "endTime": "2026-10-13T01:14:17.351146196Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460 correlation ID: 60fbb9ec-6da1-4cca-9803-0dd26c08e96f client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-2wzmp7" on minor version 4.20 Unexpected error: <exported.ResponseError \| 0x2db2723ceb40>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7 correlation ID: c3642580-1ba8-4a39-8a9e-7b6ecb2492b2 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-2wzmp7" on minor version 4.20 Unexpected error: <exported.ResponseError \| 0x2db2723ceb40>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7 correlation ID: c3642580-1ba8-4a39-8a9e-7b6ecb2492b2 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0x1d06db38a6c0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0x1d06db38a6c0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-t7fn95nw7c25/cluster-candidate-4-23-57qvrt should provision Unexpected error: <fmt.wrapError \| 0x3a5f0855d8c0>: failed to create HCP cluster cluster-candidate-4-23-57qvrt: failed waiting for cluster="cluster-candidate-4-23-57qvrt" in resourcegroup="rg-candidate-4-23-t7fn95nw7c25" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c", "name": "2705122b-6946-425e-a515-ddcaffff368c", "status": "Failed", "startTime": "2026-10-13T00:45:25.989519774Z", "endTime": "2026-10-13T01:14:27.463876895Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c correlation ID: 2a76eb98-cf1d-4901-a388-38c294f0e40f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-t7fn95nw7c25/cluster-candidate-4-23-57qvrt should provision Unexpected error: <fmt.wrapError \| 0x3a5f0855d8c0>: failed to create HCP cluster cluster-candidate-4-23-57qvrt: failed waiting for cluster="cluster-candidate-4-23-57qvrt" in resourcegroup="rg-candidate-4-23-t7fn95nw7c25" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c", "name": "2705122b-6946-425e-a515-ddcaffff368c", "status": "Failed", "startTime": "2026-10-13T00:45:25.989519774Z", "endTime": "2026-10-13T01:14:27.463876895Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c correlation ID: 2a76eb98-cf1d-4901-a388-38c294f0e40f client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0x36de7def25a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-9tkpssrgjkm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab", "name": "5b1bc7fc-bdf9-4308-a786-dfe997f22cab", "status": "Failed", "startTime": "2026-10-13T00:46:59.88326421Z", "endTime": "2026-10-13T01:16:07.327252569Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab correlation ID: 58af659d-8507-482a-af55-c3b9b34081d3 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0x36de7def25a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-9tkpssrgjkm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab", "name": "5b1bc7fc-bdf9-4308-a786-dfe997f22cab", "status": "Failed", "startTime": "2026-10-13T00:46:59.88326421Z", "endTime": "2026-10-13T01:16:07.327252569Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab correlation ID: 58af659d-8507-482a-af55-c3b9b34081d3 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0 Unexpected error: <exported.ResponseError \| 0x33ccd41339b0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0 Unexpected error: <exported.ResponseError \| 0x33ccd41339b0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23 Unexpected error: <exported.ResponseError \| 0x2299302636e0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23 Unexpected error: <exported.ResponseError \| 0x2299302636e0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21 Unexpected error: <exported.ResponseError \| 0x1206c2abaf90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2 correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21 Unexpected error: <exported.ResponseError \| 0x1206c2abaf90>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2 correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-skj8qf" on minor version 4.22 Unexpected error: <exported.ResponseError \| 0x3c62b95e41e0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf correlation ID: 60d99a02-a3a5-4d53-8c48-39141615b203 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-skj8qf" on minor version 4.22 Unexpected error: <exported.ResponseError \| 0x3c62b95e41e0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf correlation ID: 60d99a02-a3a5-4d53-8c48-39141615b203 client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0x1df4f5886960>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-52wpf8fqqws2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d", "name": "0259fdb3-2df6-41f0-b54d-4d1f2550b27d", "status": "Failed", "startTime": "2026-10-13T00:45:21.210520089Z", "endTime": "2026-10-13T01:14:27.312466771Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d correlation ID: e6c80b42-1116-4c9b-af12-981d739fb534 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0x1df4f5886960>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-52wpf8fqqws2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d", "name": "0259fdb3-2df6-41f0-b54d-4d1f2550b27d", "status": "Failed", "startTime": "2026-10-13T00:45:21.210520089Z", "endTime": "2026-10-13T01:14:27.312466771Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d correlation ID: e6c80b42-1116-4c9b-af12-981d739fb534 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-47nrhz" to 5.0 Unexpected error: <exported.ResponseError \| 0xc0b00113800>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz correlation ID: 4e563173-ee2b-41f8-bfdc-71796bcc951f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-47nrhz" to 5.0 Unexpected error: <exported.ResponseError \| 0xc0b00113800>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz correlation ID: 4e563173-ee2b-41f8-bfdc-71796bcc951f client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xpg4qn" to 4.23 Unexpected error: <exported.ResponseError \| 0x27b75529c2a0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn correlation ID: 9615e7a8-acc4-47ff-a248-a80acf2f7ffe client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xpg4qn" to 4.23 Unexpected error: <exported.ResponseError \| 0x27b75529c2a0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn correlation ID: 9615e7a8-acc4-47ff-a248-a80acf2f7ffe client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4 Unexpected error: <fmt.wrapError \| 0x3bcfe9694a80>: failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-pw28wkv425hf": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az correlation ID: b56e3add-ea82-4ecd-b4d2-ff890f4e0eba client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4 Unexpected error: <fmt.wrapError \| 0x3bcfe9694a80>: failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-pw28wkv425hf": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az correlation ID: b56e3add-ea82-4ecd-b4d2-ff890f4e0eba client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up Unexpected error: <fmt.wrapError \| 0x4249c12bcc0>: failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-q49hdkl9r8k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1 correlation ID: 70f47da6-2ca6-4f8b-b7c0-dc0e80d674d8 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up Unexpected error: <fmt.wrapError \| 0x4249c12bcc0>: failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-q49hdkl9r8k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1 correlation ID: 70f47da6-2ca6-4f8b-b7c0-dc0e80d674d8 client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x1eb12239cb20>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nmcgvph965r9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321", "name": "68342b26-1ba7-4914-8175-d1e93c601321", "status": "Failed", "startTime": "2026-10-13T00:47:22.474412187Z", "endTime": "2026-10-13T01:16:27.331795593Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321 correlation ID: 70731594-aaee-4561-aebd-de9f9ae50f08 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x1eb12239cb20>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nmcgvph965r9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321", "name": "68342b26-1ba7-4914-8175-d1e93c601321", "status": "Failed", "startTime": "2026-10-13T00:47:22.474412187Z", "endTime": "2026-10-13T01:16:27.331795593Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321 correlation ID: 70731594-aaee-4561-aebd-de9f9ae50f08 client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-cs5mpkp7bwm6/cluster-candidate-5-0-hpvbqp should provision Unexpected error: <fmt.wrapError \| 0x1732cbbf48e0>: failed to create HCP cluster cluster-candidate-5-0-hpvbqp: failed waiting for cluster="cluster-candidate-5-0-hpvbqp" in resourcegroup="rg-candidate-5-0-cs5mpkp7bwm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26", "name": "19a77831-9a94-4332-b96c-5edd63f75b26", "status": "Failed", "startTime": "2026-10-13T00:45:12.558533028Z", "endTime": "2026-10-13T01:14:17.596998966Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26 correlation ID: 72260ec2-baf6-4330-866a-41f6c7bb77ec client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-cs5mpkp7bwm6/cluster-candidate-5-0-hpvbqp should provision Unexpected error: <fmt.wrapError \| 0x1732cbbf48e0>: failed to create HCP cluster cluster-candidate-5-0-hpvbqp: failed waiting for cluster="cluster-candidate-5-0-hpvbqp" in resourcegroup="rg-candidate-5-0-cs5mpkp7bwm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26", "name": "19a77831-9a94-4332-b96c-5edd63f75b26", "status": "Failed", "startTime": "2026-10-13T00:45:12.558533028Z", "endTime": "2026-10-13T01:14:17.596998966Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26 correlation ID: 72260ec2-baf6-4330-866a-41f6c7bb77ec client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision Unexpected error: <fmt.wrapError \| 0x15168f06cae0>: failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757", "name": "8479abf5-39da-412b-bb04-87374bf3b757", "status": "Failed", "startTime": "2026-10-13T00:45:19.685007768Z", "endTime": "2026-10-13T01:14:27.351105107Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757 correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision Unexpected error: <fmt.wrapError \| 0x15168f06cae0>: failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757", "name": "8479abf5-39da-412b-bb04-87374bf3b757", "status": "Failed", "startTime": "2026-10-13T00:45:19.685007768Z", "endTime": "2026-10-13T01:14:27.351105107Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757 correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0x1854e4898c30>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0x1854e4898c30>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-n7whnh" to 4.21 Unexpected error: <exported.ResponseError \| 0x2dbf65a18e10>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh correlation ID: 7632d777-eaad-4533-834b-b2838b456b70 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-n7whnh" to 4.21 Unexpected error: <exported.ResponseError \| 0x2dbf65a18e10>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh correlation ID: 7632d777-eaad-4533-834b-b2838b456b70 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7s9zwv" to 4.22 Unexpected error: <exported.ResponseError \| 0x336e294660c0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv correlation ID: 87a3049c-e745-4350-8cb4-9cce65418888 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7s9zwv" to 4.22 Unexpected error: <exported.ResponseError \| 0x336e294660c0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv correlation ID: 87a3049c-e745-4350-8cb4-9cce65418888 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0x276a4dc0cde0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster correlation ID: 4e9943d8-58b7-4b59-b33f-7b4d1323766d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0x276a4dc0cde0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster correlation ID: 4e9943d8-58b7-4b59-b33f-7b4d1323766d client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pfwvc4jbpftc/cluster-candidate-4-20-7b9hxf should provision Unexpected error: <fmt.wrapError \| 0x3001f81fa6e0>: failed to create HCP cluster cluster-candidate-4-20-7b9hxf: failed waiting for cluster="cluster-candidate-4-20-7b9hxf" in resourcegroup="rg-candidate-4-20-pfwvc4jbpftc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c", "name": "a72b40eb-b9a6-4461-8444-590e0d43a12c", "status": "Failed", "startTime": "2026-10-13T00:45:10.329847646Z", "endTime": "2026-10-13T01:14:17.357102947Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c correlation ID: 0e41648f-f7cc-40e7-90b9-27570cd2acfd client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pfwvc4jbpftc/cluster-candidate-4-20-7b9hxf should provision Unexpected error: <fmt.wrapError \| 0x3001f81fa6e0>: failed to create HCP cluster cluster-candidate-4-20-7b9hxf: failed waiting for cluster="cluster-candidate-4-20-7b9hxf" in resourcegroup="rg-candidate-4-20-pfwvc4jbpftc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c", "name": "a72b40eb-b9a6-4461-8444-590e0d43a12c", "status": "Failed", "startTime": "2026-10-13T00:45:10.329847646Z", "endTime": "2026-10-13T01:14:17.357102947Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c correlation ID: 0e41648f-f7cc-40e7-90b9-27570cd2acfd client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35 Unexpected error: <fmt.wrapError \| 0x321c1d43a020>: failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-tj8nnc" resourcegroup="rg-np-downgrade-tj8nnc-l7vlmpmg9zkq": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21 correlation ID: 9ba8d957-0858-4dfe-843e-e866942b6301 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35 Unexpected error: <fmt.wrapError \| 0x321c1d43a020>: failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-tj8nnc" resourcegroup="rg-np-downgrade-tj8nnc-l7vlmpmg9zkq": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21 correlation ID: 9ba8d957-0858-4dfe-843e-e866942b6301 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36 Unexpected error: <fmt.wrapError \| 0x11c33420360>: failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-8dtxpr" resourcegroup="rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20 correlation ID: 5727eda3-77c1-48f6-a819-63329d1fe751 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36 Unexpected error: <fmt.wrapError \| 0x11c33420360>: failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-8dtxpr" resourcegroup="rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20 correlation ID: 5727eda3-77c1-48f6-a819-63329d1fe751 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42 Unexpected error: <fmt.wrapError \| 0x31faef804060>: failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20 correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42 Unexpected error: <fmt.wrapError \| 0x31faef804060>: failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20 correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36 Unexpected error: <fmt.wrapError \| 0x258acfd8b940>: failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21 correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36 Unexpected error: <fmt.wrapError \| 0x258acfd8b940>: failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21 correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.41 to 4.20.42 Unexpected error: <fmt.wrapError \| 0x1aca3ec93b80>: failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-dnzbj8" resourcegroup="rg-np-noedge-dnzbj8-g252879f889l": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20 correlation ID: 6ca4b7ae-3d17-4f46-bfcb-49c19eef2097 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.41 to 4.20.42 Unexpected error: <fmt.wrapError \| 0x1aca3ec93b80>: failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-dnzbj8" resourcegroup="rg-np-noedge-dnzbj8-g252879f889l": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20 correlation ID: 6ca4b7ae-3d17-4f46-bfcb-49c19eef2097 client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0x15a1e46902c0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346", "name": "8f006a7a-53bf-4566-93d0-8202c11da346", "status": "Failed", "startTime": "2026-10-13T00:52:36.6753625Z", "endTime": "2026-10-13T01:21:37.350837076Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346 correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0x15a1e46902c0>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346", "name": "8f006a7a-53bf-4566-93d0-8202c11da346", "status": "Failed", "startTime": "2026-10-13T00:52:36.6753625Z", "endTime": "2026-10-13T01:21:37.350837076Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346 correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777 client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1 Unexpected error: <exported.ResponseError \| 0x13a151ae2450>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp correlation ID: b7196ccc-8430-45a5-9628-149904d927cb client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1 Unexpected error: <exported.ResponseError \| 0x13a151ae2450>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.cluster.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp correlation ID: b7196ccc-8430-45a5-9628-149904d927cb client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42 Unexpected error: <fmt.wrapError \| 0x1b553dfaa40>: failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420 correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42 Unexpected error: <fmt.wrapError \| 0x1b553dfaa40>: failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420 correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f client request ID: unavailable occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.42 to 4.22.18 Unexpected error: <fmt.wrapError \| 0xbf6d3432b60>: failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-vf6ldk" resourcegroup="rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422 correlation ID: 8ad4ceee-185b-404d-b4d6-7323c6c6cfb5 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.42 to 4.22.18 Unexpected error: <fmt.wrapError \| 0xbf6d3432b60>: failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-vf6ldk" resourcegroup="rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag", "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422 correlation ID: 8ad4ceee-185b-404d-b4d6-7323c6c6cfb5 client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 00:11:54 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x3152b3c1a500>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c", "name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c", "status": "Failed", "startTime": "2026-10-13T00:45:30.036006045Z", "endTime": "2026-10-13T01:14:37.44566298Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x3152b3c1a500>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c", "name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c", "status": "Failed", "startTime": "2026-10-13T00:45:30.036006045Z", "endTime": "2026-10-13T01:14:37.44566298Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8 client request ID: unavailable occurred

Time (UTC)

Runtime

Job

Region

Failed at

Result

Failed tests

Details

23:00:48

1h 47m 24s

periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly

post-goodmerged PR

No additional run metadata captured

uksouth

e2e

failed

n/a

e2e (6)

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

23:00:48 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x272ecda2fb90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 3b21cb42-ff17-49b5-9ed9-e3d69922540a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x272ecda2fb90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-h99vq8np96rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 3b21cb42-ff17-49b5-9ed9-e3d69922540a
            client request ID: unavailable
occurred

23:00:48 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3bf246e81b90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: c883dc6d-a815-4804-970f-295c17dcee98
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3bf246e81b90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-kdzwfbjg55gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: c883dc6d-a815-4804-970f-295c17dcee98
            client request ID: unavailable
occurred

23:00:48 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x25636a4bb950>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: eb7e7c6f-c7e9-46a0-8d7d-c420ec91eb1d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x25636a4bb950>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nh5qcpqmqsm8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: eb7e7c6f-c7e9-46a0-8d7d-c420ec91eb1d
            client request ID: unavailable
occurred

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift

23:00:48 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-kxcsmnps9zfz/cluster-nightly-4-21-j9wst5 should provision
Unexpected error:
    <*fmt.wrapError | 0x21388a648d60>: 
    failed to create HCP cluster cluster-nightly-4-21-j9wst5: failed waiting for cluster="cluster-nightly-4-21-j9wst5" in resourcegroup="rg-nightly-4-21-kxcsmnps9zfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d",
      "name": "6943378a-a2ac-4799-849e-25f8e97d936d",
      "status": "Failed",
      "startTime": "2026-10-13T23:24:54.16266447Z",
      "endTime": "2026-10-13T23:54:01.266968226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d
            correlation ID: 11b5d2e6-bd60-46f6-9cea-eb83bb792173
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-kxcsmnps9zfz/cluster-nightly-4-21-j9wst5 should provision
Unexpected error:
    <*fmt.wrapError | 0x21388a648d60>: 
    failed to create HCP cluster cluster-nightly-4-21-j9wst5: failed waiting for cluster="cluster-nightly-4-21-j9wst5" in resourcegroup="rg-nightly-4-21-kxcsmnps9zfz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d",
      "name": "6943378a-a2ac-4799-849e-25f8e97d936d",
      "status": "Failed",
      "startTime": "2026-10-13T23:24:54.16266447Z",
      "endTime": "2026-10-13T23:54:01.266968226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6943378a-a2ac-4799-849e-25f8e97d936d
            correlation ID: 11b5d2e6-bd60-46f6-9cea-eb83bb792173
            client request ID: unavailable
occurred

23:00:48 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x1fbd9ed3dd40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 3c34a173-f084-4c74-8bf8-8d8da2ea7e20
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x1fbd9ed3dd40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-69wnspjznlc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 3c34a173-f084-4c74-8bf8-8d8da2ea7e20
            client request ID: unavailable
occurred

23:00:48 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x1ca0706e2750>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: dd69bc64-32de-4cf3-a00f-c421bb664b95
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x1ca0706e2750>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-8fp5rtv4jcsv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: dd69bc64-32de-4cf3-a00f-c421bb664b95
            client request ID: unavailable
occurred

00:11:54

7h 10m 8s

branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel

post-goodmerged PR

No additional run metadata captured

westus3

e2e

failed

n/a

e2e (33)

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-dwlbs8wqmf4n/cluster-candidate-5-1-dhmlhx should provision
Unexpected error:
    <*fmt.wrapError | 0x32444a40800>: 
    failed to create HCP cluster cluster-candidate-5-1-dhmlhx: failed waiting for cluster="cluster-candidate-5-1-dhmlhx" in resourcegroup="rg-candidate-5-1-dwlbs8wqmf4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8",
      "name": "cca958a7-cdd6-4407-a240-933a8c20efd8",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:11.941637821Z",
      "endTime": "2026-10-13T01:14:17.337870809Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8
            correlation ID: 4827029f-0d94-4b20-83b7-9db0cfabf792
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-dwlbs8wqmf4n/cluster-candidate-5-1-dhmlhx should provision
Unexpected error:
    <*fmt.wrapError | 0x32444a40800>: 
    failed to create HCP cluster cluster-candidate-5-1-dhmlhx: failed waiting for cluster="cluster-candidate-5-1-dhmlhx" in resourcegroup="rg-candidate-5-1-dwlbs8wqmf4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8",
      "name": "cca958a7-cdd6-4407-a240-933a8c20efd8",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:11.941637821Z",
      "endTime": "2026-10-13T01:14:17.337870809Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca958a7-cdd6-4407-a240-933a8c20efd8
            correlation ID: 4827029f-0d94-4b20-83b7-9db0cfabf792
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-24hzxh9hvdbr/cluster-candidate-4-22-dg2r6h should provision
Unexpected error:
    <*fmt.wrapError | 0x1893baafcc80>: 
    failed to create HCP cluster cluster-candidate-4-22-dg2r6h: failed waiting for cluster="cluster-candidate-4-22-dg2r6h" in resourcegroup="rg-candidate-4-22-24hzxh9hvdbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460",
      "name": "7a5573b4-6d8c-4cfa-88d0-da0d2af71460",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:09.767142622Z",
      "endTime": "2026-10-13T01:14:17.351146196Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460
            correlation ID: 60fbb9ec-6da1-4cca-9803-0dd26c08e96f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-24hzxh9hvdbr/cluster-candidate-4-22-dg2r6h should provision
Unexpected error:
    <*fmt.wrapError | 0x1893baafcc80>: 
    failed to create HCP cluster cluster-candidate-4-22-dg2r6h: failed waiting for cluster="cluster-candidate-4-22-dg2r6h" in resourcegroup="rg-candidate-4-22-24hzxh9hvdbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460",
      "name": "7a5573b4-6d8c-4cfa-88d0-da0d2af71460",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:09.767142622Z",
      "endTime": "2026-10-13T01:14:17.351146196Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a5573b4-6d8c-4cfa-88d0-da0d2af71460
            correlation ID: 60fbb9ec-6da1-4cca-9803-0dd26c08e96f
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-2wzmp7" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x2db2723ceb40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7
            correlation ID: c3642580-1ba8-4a39-8a9e-7b6ecb2492b2
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-2wzmp7" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x2db2723ceb40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-b5v8fzd2jhjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2wzmp7
            correlation ID: c3642580-1ba8-4a39-8a9e-7b6ecb2492b2
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x1d06db38a6c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x1d06db38a6c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-t7fn95nw7c25/cluster-candidate-4-23-57qvrt should provision
Unexpected error:
    <*fmt.wrapError | 0x3a5f0855d8c0>: 
    failed to create HCP cluster cluster-candidate-4-23-57qvrt: failed waiting for cluster="cluster-candidate-4-23-57qvrt" in resourcegroup="rg-candidate-4-23-t7fn95nw7c25" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c",
      "name": "2705122b-6946-425e-a515-ddcaffff368c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:25.989519774Z",
      "endTime": "2026-10-13T01:14:27.463876895Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c
            correlation ID: 2a76eb98-cf1d-4901-a388-38c294f0e40f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-t7fn95nw7c25/cluster-candidate-4-23-57qvrt should provision
Unexpected error:
    <*fmt.wrapError | 0x3a5f0855d8c0>: 
    failed to create HCP cluster cluster-candidate-4-23-57qvrt: failed waiting for cluster="cluster-candidate-4-23-57qvrt" in resourcegroup="rg-candidate-4-23-t7fn95nw7c25" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c",
      "name": "2705122b-6946-425e-a515-ddcaffff368c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:25.989519774Z",
      "endTime": "2026-10-13T01:14:27.463876895Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2705122b-6946-425e-a515-ddcaffff368c
            correlation ID: 2a76eb98-cf1d-4901-a388-38c294f0e40f
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x36de7def25a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-9tkpssrgjkm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab",
      "name": "5b1bc7fc-bdf9-4308-a786-dfe997f22cab",
      "status": "Failed",
      "startTime": "2026-10-13T00:46:59.88326421Z",
      "endTime": "2026-10-13T01:16:07.327252569Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab
            correlation ID: 58af659d-8507-482a-af55-c3b9b34081d3
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x36de7def25a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-9tkpssrgjkm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab",
      "name": "5b1bc7fc-bdf9-4308-a786-dfe997f22cab",
      "status": "Failed",
      "startTime": "2026-10-13T00:46:59.88326421Z",
      "endTime": "2026-10-13T01:16:07.327252569Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5b1bc7fc-bdf9-4308-a786-dfe997f22cab
            correlation ID: 58af659d-8507-482a-af55-c3b9b34081d3
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x33ccd41339b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
            correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x33ccd41339b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
            correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x2299302636e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
            correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x2299302636e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
            correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x1206c2abaf90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
            correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x1206c2abaf90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
            correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-skj8qf" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x3c62b95e41e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf
            correlation ID: 60d99a02-a3a5-4d53-8c48-39141615b203
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-skj8qf" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x3c62b95e41e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-tggxjftkrj47/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-skj8qf
            correlation ID: 60d99a02-a3a5-4d53-8c48-39141615b203
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1df4f5886960>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-52wpf8fqqws2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d",
      "name": "0259fdb3-2df6-41f0-b54d-4d1f2550b27d",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:21.210520089Z",
      "endTime": "2026-10-13T01:14:27.312466771Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d
            correlation ID: e6c80b42-1116-4c9b-af12-981d739fb534
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1df4f5886960>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-52wpf8fqqws2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d",
      "name": "0259fdb3-2df6-41f0-b54d-4d1f2550b27d",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:21.210520089Z",
      "endTime": "2026-10-13T01:14:27.312466771Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0259fdb3-2df6-41f0-b54d-4d1f2550b27d
            correlation ID: e6c80b42-1116-4c9b-af12-981d739fb534
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-47nrhz" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0b00113800>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz
            correlation ID: 4e563173-ee2b-41f8-bfdc-71796bcc951f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-47nrhz" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0b00113800>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-2ljqrs9zmbz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-47nrhz
            correlation ID: 4e563173-ee2b-41f8-bfdc-71796bcc951f
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xpg4qn" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x27b75529c2a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn
            correlation ID: 9615e7a8-acc4-47ff-a248-a80acf2f7ffe
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xpg4qn" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x27b75529c2a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-48cmz5s24f6t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xpg4qn
            correlation ID: 9615e7a8-acc4-47ff-a248-a80acf2f7ffe
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x3bcfe9694a80>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-pw28wkv425hf": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
            correlation ID: b56e3add-ea82-4ecd-b4d2-ff890f4e0eba
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x3bcfe9694a80>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-pw28wkv425hf": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pw28wkv425hf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
            correlation ID: b56e3add-ea82-4ecd-b4d2-ff890f4e0eba
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x4249c12bcc0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-q49hdkl9r8k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
            correlation ID: 70f47da6-2ca6-4f8b-b7c0-dc0e80d674d8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x4249c12bcc0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-q49hdkl9r8k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-q49hdkl9r8k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
            correlation ID: 70f47da6-2ca6-4f8b-b7c0-dc0e80d674d8
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x1eb12239cb20>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nmcgvph965r9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321",
      "name": "68342b26-1ba7-4914-8175-d1e93c601321",
      "status": "Failed",
      "startTime": "2026-10-13T00:47:22.474412187Z",
      "endTime": "2026-10-13T01:16:27.331795593Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321
            correlation ID: 70731594-aaee-4561-aebd-de9f9ae50f08
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x1eb12239cb20>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nmcgvph965r9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321",
      "name": "68342b26-1ba7-4914-8175-d1e93c601321",
      "status": "Failed",
      "startTime": "2026-10-13T00:47:22.474412187Z",
      "endTime": "2026-10-13T01:16:27.331795593Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/68342b26-1ba7-4914-8175-d1e93c601321
            correlation ID: 70731594-aaee-4561-aebd-de9f9ae50f08
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-cs5mpkp7bwm6/cluster-candidate-5-0-hpvbqp should provision
Unexpected error:
    <*fmt.wrapError | 0x1732cbbf48e0>: 
    failed to create HCP cluster cluster-candidate-5-0-hpvbqp: failed waiting for cluster="cluster-candidate-5-0-hpvbqp" in resourcegroup="rg-candidate-5-0-cs5mpkp7bwm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26",
      "name": "19a77831-9a94-4332-b96c-5edd63f75b26",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:12.558533028Z",
      "endTime": "2026-10-13T01:14:17.596998966Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26
            correlation ID: 72260ec2-baf6-4330-866a-41f6c7bb77ec
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-cs5mpkp7bwm6/cluster-candidate-5-0-hpvbqp should provision
Unexpected error:
    <*fmt.wrapError | 0x1732cbbf48e0>: 
    failed to create HCP cluster cluster-candidate-5-0-hpvbqp: failed waiting for cluster="cluster-candidate-5-0-hpvbqp" in resourcegroup="rg-candidate-5-0-cs5mpkp7bwm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26",
      "name": "19a77831-9a94-4332-b96c-5edd63f75b26",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:12.558533028Z",
      "endTime": "2026-10-13T01:14:17.596998966Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a77831-9a94-4332-b96c-5edd63f75b26
            correlation ID: 72260ec2-baf6-4330-866a-41f6c7bb77ec
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
    <*fmt.wrapError | 0x15168f06cae0>: 
    failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
      "name": "8479abf5-39da-412b-bb04-87374bf3b757",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:19.685007768Z",
      "endTime": "2026-10-13T01:14:27.351105107Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
            correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
    <*fmt.wrapError | 0x15168f06cae0>: 
    failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
      "name": "8479abf5-39da-412b-bb04-87374bf3b757",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:19.685007768Z",
      "endTime": "2026-10-13T01:14:27.351105107Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
            correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1854e4898c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1854e4898c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-n7whnh" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x2dbf65a18e10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh
            correlation ID: 7632d777-eaad-4533-834b-b2838b456b70
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-n7whnh" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x2dbf65a18e10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-s7npn22q29ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-n7whnh
            correlation ID: 7632d777-eaad-4533-834b-b2838b456b70
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7s9zwv" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x336e294660c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv
            correlation ID: 87a3049c-e745-4350-8cb4-9cce65418888
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7s9zwv" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x336e294660c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-9t8jvmnxhvhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7s9zwv
            correlation ID: 87a3049c-e745-4350-8cb4-9cce65418888
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x276a4dc0cde0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 4e9943d8-58b7-4b59-b33f-7b4d1323766d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x276a4dc0cde0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zmjd7m89bwr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 4e9943d8-58b7-4b59-b33f-7b4d1323766d
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pfwvc4jbpftc/cluster-candidate-4-20-7b9hxf should provision
Unexpected error:
    <*fmt.wrapError | 0x3001f81fa6e0>: 
    failed to create HCP cluster cluster-candidate-4-20-7b9hxf: failed waiting for cluster="cluster-candidate-4-20-7b9hxf" in resourcegroup="rg-candidate-4-20-pfwvc4jbpftc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c",
      "name": "a72b40eb-b9a6-4461-8444-590e0d43a12c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:10.329847646Z",
      "endTime": "2026-10-13T01:14:17.357102947Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c
            correlation ID: 0e41648f-f7cc-40e7-90b9-27570cd2acfd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-pfwvc4jbpftc/cluster-candidate-4-20-7b9hxf should provision
Unexpected error:
    <*fmt.wrapError | 0x3001f81fa6e0>: 
    failed to create HCP cluster cluster-candidate-4-20-7b9hxf: failed waiting for cluster="cluster-candidate-4-20-7b9hxf" in resourcegroup="rg-candidate-4-20-pfwvc4jbpftc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c",
      "name": "a72b40eb-b9a6-4461-8444-590e0d43a12c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:10.329847646Z",
      "endTime": "2026-10-13T01:14:17.357102947Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a72b40eb-b9a6-4461-8444-590e0d43a12c
            correlation ID: 0e41648f-f7cc-40e7-90b9-27570cd2acfd
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x321c1d43a020>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-tj8nnc" resourcegroup="rg-np-downgrade-tj8nnc-l7vlmpmg9zkq": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21
            correlation ID: 9ba8d957-0858-4dfe-843e-e866942b6301
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x321c1d43a020>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-tj8nnc" resourcegroup="rg-np-downgrade-tj8nnc-l7vlmpmg9zkq": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-tj8nnc-l7vlmpmg9zkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-tj8nnc/nodePools/npdg-4-21
            correlation ID: 9ba8d957-0858-4dfe-843e-e866942b6301
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x11c33420360>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-8dtxpr" resourcegroup="rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20
            correlation ID: 5727eda3-77c1-48f6-a819-63329d1fe751
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x11c33420360>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-8dtxpr" resourcegroup="rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8dtxpr-8nnjrwtmtkct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8dtxpr/nodePools/npupgrade-4-20
            correlation ID: 5727eda3-77c1-48f6-a819-63329d1fe751
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x31faef804060>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
            correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x31faef804060>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
            correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x258acfd8b940>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
            correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x258acfd8b940>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
            correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.41 to 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1aca3ec93b80>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-dnzbj8" resourcegroup="rg-np-noedge-dnzbj8-g252879f889l": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20
            correlation ID: 6ca4b7ae-3d17-4f46-bfcb-49c19eef2097
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.41 to 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1aca3ec93b80>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-dnzbj8" resourcegroup="rg-np-noedge-dnzbj8-g252879f889l": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dnzbj8-g252879f889l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-dnzbj8/nodePools/npnoedge-4-20
            correlation ID: 6ca4b7ae-3d17-4f46-bfcb-49c19eef2097
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x15a1e46902c0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
      "name": "8f006a7a-53bf-4566-93d0-8202c11da346",
      "status": "Failed",
      "startTime": "2026-10-13T00:52:36.6753625Z",
      "endTime": "2026-10-13T01:21:37.350837076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
            correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x15a1e46902c0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
      "name": "8f006a7a-53bf-4566-93d0-8202c11da346",
      "status": "Failed",
      "startTime": "2026-10-13T00:52:36.6753625Z",
      "endTime": "2026-10-13T01:21:37.350837076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
            correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x13a151ae2450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
            correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x13a151ae2450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
            correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1b553dfaa40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
            correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1b553dfaa40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
            correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
            client request ID: unavailable
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift

00:11:54 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.42 to 4.22.18
Unexpected error:
    <*fmt.wrapError | 0xbf6d3432b60>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-vf6ldk" resourcegroup="rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422
            correlation ID: 8ad4ceee-185b-404d-b4d6-7323c6c6cfb5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.42 to 4.22.18
Unexpected error:
    <*fmt.wrapError | 0xbf6d3432b60>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-vf6ldk" resourcegroup="rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vf6ldk-m7qsdk9bdhfd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vf6ldk/nodePools/nps-420-422
            correlation ID: 8ad4ceee-185b-404d-b4d6-7323c6c6cfb5
            client request ID: unavailable
occurred

00:11:54 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3152b3c1a500>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:30.036006045Z",
      "endTime": "2026-10-13T01:14:37.44566298Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
            correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3152b3c1a500>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
      "status": "Failed",
      "startTime": "2026-10-13T00:45:30.036006045Z",
      "endTime": "2026-10-13T01:14:37.44566298Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
            correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
            client request ID: unavailable
occurred