CIHealth Run Log

Environment: STG

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:00:08	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	10	e2e (10) UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded 23:00:08 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:228]: failed to update node pool np-1 with new label and scale up Unexpected error: <fmt.wrapErrors \| 0xc002a1d1a0>: failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-l7xnjldf8s7s" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-l7xnjldf8s7s, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:228]: failed to update node pool np-1 with new label and scale up Unexpected error: <fmt.wrapErrors \| 0xc002a1d1a0>: failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-l7xnjldf8s7s" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-l7xnjldf8s7s, error: context deadline exceeded ... occurred list nodes (nodePool=<nodepool>): Get "<url> 23:00:08 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.000s. Expected success, but got an error: <fmt.wrapError \| 0xc0010e2920>: list nodes (nodePool=nps-420-422): Get "https://api.m1w5a4x7b2n1l4i.t4j1.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": EOF ... fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.000s. Expected success, but got an error: <fmt.wrapError \| 0xc0010e2920>: list nodes (nodePool=nps-420-422): Get "https://api.m1w5a4x7b2n1l4i.t4j1.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": EOF ... VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "<url> 23:00:08 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.007s. Expected success, but got an error: <errors.joinError \| 0xc000fcc5b8>: VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-20-065923, targetMinor=4.22.0-0.nightly-multi-2026-08-20-141954) failed: failed to get clusterversion "version": Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout VerifyBasicAccess failed: failed to list services: Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.007s. Expected success, but got an error: <errors.joinError \| 0xc000fcc5b8>: VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-20-065923, targetMinor=4.22.0-0.nightly-multi-2026-08-20-141954) failed: failed to get clusterversion "version": Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout VerifyBasicAccess failed: failed to list services: Get "https://api.u0i9f7m5k4k1z6j.gvbd.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout ... tls: failed to verify certificate: x509: certificate signed by unknown authority 23:00:08 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:336]: failed to run simple web app and connectivity check app with cilium CNI Unexpected error: <errors.joinError \| 0xc0025d2270>: VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-7lgrn.apps.aro.agg-cluster.zfzg.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:336]: failed to run simple web app and connectivity check app with cilium CNI Unexpected error: <errors.joinError \| 0xc0025d2270>: VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-7lgrn.apps.aro.agg-cluster.zfzg.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority ... occurred VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url> 23:00:08 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.006s. Expected success, but got an error: <errors.joinError \| 0xc000f0c138>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22.0-0.nightly-multi-2026-08-20-141954, targetMinor=4.23.0-0.nightly-multi-2026-08-20-085020) failed: failed to get clusterversion "version": Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout VerifyBasicAccess failed: failed to list services: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.006s. Expected success, but got an error: <errors.joinError \| 0xc000f0c138>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22.0-0.nightly-multi-2026-08-20-141954, targetMinor=4.23.0-0.nightly-multi-2026-08-20-085020) failed: failed to get clusterversion "version": Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout VerifyBasicAccess failed: failed to list services: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout ... UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded 23:00:08 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:196]: failed to upgrade node pool npupgrade-4-20 to version 4.21.0-0.nightly-multi-2026-08-20-065923 Unexpected error: <fmt.wrapErrors \| 0xc002d08a80>: failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-z5466s" resourcegroup="rg-np-version-upgrade-z5466s-wkz757zpgbsr" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-z5466s in resource group rg-np-version-upgrade-z5466s-wkz757zpgbsr, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:196]: failed to upgrade node pool npupgrade-4-20 to version 4.21.0-0.nightly-multi-2026-08-20-065923 Unexpected error: <fmt.wrapErrors \| 0xc002d08a80>: failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-z5466s" resourcegroup="rg-np-version-upgrade-z5466s-wkz757zpgbsr" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-z5466s in resource group rg-np-version-upgrade-z5466s-wkz757zpgbsr, error: context deadline exceeded ... occurred list nodes (nodePool=<nodepool>): Get "<url> 23:00:08 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:772]: Timed out after 2700.000s. node pool nodes were not recreated at downgrade target version 4.20.0-0.nightly-multi-2026-08-20-052229 Expected success, but got an error: <fmt.wrapError \| 0xc000df20a0>: list nodes (nodePool=npdg-422-420): Get "https://api.d1i3c9v1k0g4b5v.7gi9.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout ... fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:772]: Timed out after 2700.000s. node pool nodes were not recreated at downgrade target version 4.20.0-0.nightly-multi-2026-08-20-052229 Expected success, but got an error: <fmt.wrapError \| 0xc000df20a0>: list nodes (nodePool=npdg-422-420): Get "https://api.d1i3c9v1k0g4b5v.7gi9.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout ... VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url> 23:00:08 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s. Expected success, but got an error: <errors.joinError \| 0xc000b06150>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout VerifyBasicAccess failed: failed to list services: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-20-052229, targetMinor=4.21.0-0.nightly-multi-2026-08-20-065923) failed: failed to get clusterversion "version": Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s. Expected success, but got an error: <errors.joinError \| 0xc000b06150>: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout VerifyBasicAccess failed: failed to list services: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-20-052229, targetMinor=4.21.0-0.nightly-multi-2026-08-20-065923) failed: failed to get clusterversion "version": Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout ... timeout during CreateHCPClusterAndWait; context deadline exceeded 23:00:08 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapErrors \| 0xc0027ec2a0>: failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-cxh78blbgmpq, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-cxh78blbgmpq" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-cxh78blbgmpq, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapErrors \| 0xc0027ec2a0>: failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-cxh78blbgmpq, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-cxh78blbgmpq" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-cxh78blbgmpq, error: context deadline exceeded ... occurred VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5 23:00:08 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.008s. Expected success, but got an error: <errors.joinError \| 0xc0013e8a98>: VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-08-20-024947, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: failed to get clusterversion "version": Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": EOF VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": EOF VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/version": EOF VerifyBasicAccess failed: failed to list services: Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": EOF ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.008s. Expected success, but got an error: <errors.joinError \| 0xc0013e8a98>: VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-08-20-024947, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: failed to get clusterversion "version": Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": EOF VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": EOF VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/version": EOF VerifyBasicAccess failed: failed to list services: Get "https://api.v7t4t4y8f3t0a2n.l1x0.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": EOF ...
20:36:34	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 20:36:34 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-pt5scqn6n82h/cluster-candidate-4-21-28flhq should provision Unexpected error: <fmt.wrapError \| 0xc00124eb60>: failed to create HCP cluster cluster-candidate-4-21-28flhq: failed waiting for cluster="cluster-candidate-4-21-28flhq" in resourcegroup="rg-candidate-4-21-pt5scqn6n82h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc", "name": "332f74a8-d0dc-4e4a-9737-70119dac90bc", "status": "Failed", "startTime": "2026-08-20T20:53:42.523371474Z", "endTime": "2026-08-20T21:12:49.332578125Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-pt5scqn6n82h/cluster-candidate-4-21-28flhq should provision Unexpected error: <fmt.wrapError \| 0xc00124eb60>: failed to create HCP cluster cluster-candidate-4-21-28flhq: failed waiting for cluster="cluster-candidate-4-21-28flhq" in resourcegroup="rg-candidate-4-21-pt5scqn6n82h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc", "name": "332f74a8-d0dc-4e4a-9737-70119dac90bc", "status": "Failed", "startTime": "2026-08-20T20:53:42.523371474Z", "endTime": "2026-08-20T21:12:49.332578125Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred
12:36:44	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	6	e2e (6) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 12:36:44 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000fe3060>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-8vkz6566fxqz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/140c4b91-4a5b-4a44-a63d-5f39fc772c32 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/140c4b91-4a5b-4a44-a63d-5f39fc772c32", "name": "140c4b91-4a5b-4a44-a63d-5f39fc772c32", "status": "Failed", "startTime": "2026-08-20T13:16:33.753602969Z", "endTime": "2026-08-20T13:35:40.453844587Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-autoscaler, cluster-node-tuning-operator, redhat-operators-catalog, certified-operators-catalog, kube-controller-manager, azure-cloud-controller-manager, cluster-network-operator, packageserver, olm-collect-profiles, redhat-marketplace-catalog, openshift-apiserver, dns-operator, catalog-operator, olm-operator, cluster-policy-controller, cluster-storage-operator, machine-approver, openshift-controller-manager, hosted-cluster-config-operator, ingress-operator, cluster-image-registry-operator, konnectivity-agent, kube-storage-version-migrator, kube-scheduler, openshift-route-controller-manager, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000fe3060>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-8vkz6566fxqz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/140c4b91-4a5b-4a44-a63d-5f39fc772c32 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/140c4b91-4a5b-4a44-a63d-5f39fc772c32", "name": "140c4b91-4a5b-4a44-a63d-5f39fc772c32", "status": "Failed", "startTime": "2026-08-20T13:16:33.753602969Z", "endTime": "2026-08-20T13:35:40.453844587Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-autoscaler, cluster-node-tuning-operator, redhat-operators-catalog, certified-operators-catalog, kube-controller-manager, azure-cloud-controller-manager, cluster-network-operator, packageserver, olm-collect-profiles, redhat-marketplace-catalog, openshift-apiserver, dns-operator, catalog-operator, olm-operator, cluster-policy-controller, cluster-storage-operator, machine-approver, openshift-controller-manager, hosted-cluster-config-operator, ingress-operator, cluster-image-registry-operator, konnectivity-agent, kube-storage-version-migrator, kube-scheduler, openshift-route-controller-manager, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 12:36:44 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-4r45c4hp7cql/cluster-candidate-4-20-r96znq should provision Unexpected error: <fmt.wrapError \| 0xc0008fa700>: failed to create HCP cluster cluster-candidate-4-20-r96znq: failed waiting for cluster="cluster-candidate-4-20-r96znq" in resourcegroup="rg-candidate-4-20-4r45c4hp7cql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e", "name": "3742f6f7-818c-4bbe-a0c4-ca70030eb73e", "status": "Failed", "startTime": "2026-08-20T13:15:18.696546382Z", "endTime": "2026-08-20T13:34:20.875186386Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-4r45c4hp7cql/cluster-candidate-4-20-r96znq should provision Unexpected error: <fmt.wrapError \| 0xc0008fa700>: failed to create HCP cluster cluster-candidate-4-20-r96znq: failed waiting for cluster="cluster-candidate-4-20-r96znq" in resourcegroup="rg-candidate-4-20-4r45c4hp7cql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e", "name": "3742f6f7-818c-4bbe-a0c4-ca70030eb73e", "status": "Failed", "startTime": "2026-08-20T13:15:18.696546382Z", "endTime": "2026-08-20T13:34:20.875186386Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 12:36:44 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0011f24f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listwvkfxb" in resourcegroup="ea-list-q8zktqx9k8tx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050", "name": "7e8f24bc-6934-4503-b388-7fea73015050", "status": "Failed", "startTime": "2026-08-20T14:04:49.548371908Z", "endTime": "2026-08-20T14:28:52.367967433Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sabeurund3r70lf12nj56m098p5j7j9 still exists (deletion dispatched at 2026-08-20T14:04:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc0011f24f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listwvkfxb" in resourcegroup="ea-list-q8zktqx9k8tx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050", "name": "7e8f24bc-6934-4503-b388-7fea73015050", "status": "Failed", "startTime": "2026-08-20T14:04:49.548371908Z", "endTime": "2026-08-20T14:28:52.367967433Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sabeurund3r70lf12nj56m098p5j7j9 still exists (deletion dispatched at 2026-08-20T14:04:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred tls: failed to verify certificate: x509: certificate signed by unknown authority 12:36:44 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:336]: failed to run simple web app and connectivity check app with cilium CNI Unexpected error: <errors.joinError \| 0xc000de51e8>: VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-2lhgp.apps.aro.agg-cluster.v7n3.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:336]: failed to run simple web app and connectivity check app with cilium CNI Unexpected error: <errors.joinError \| 0xc000de51e8>: VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-2lhgp.apps.aro.agg-cluster.v7n3.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 12:36:44 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000dc4980>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-2qbp55nw5hq6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3cab7363-1a89-4114-8a30-15cd32af7525 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3cab7363-1a89-4114-8a30-15cd32af7525", "name": "3cab7363-1a89-4114-8a30-15cd32af7525", "status": "Failed", "startTime": "2026-08-20T13:17:04.908245766Z", "endTime": "2026-08-20T13:36:10.801854971Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, konnectivity-agent, catalog-operator, packageserver, redhat-operators-catalog, community-operators-catalog, kube-controller-manager, cluster-version-operator, openshift-controller-manager, cluster-policy-controller, azure-cloud-controller-manager, olm-operator, kube-scheduler, dns-operator, ingress-operator, csi-snapshot-controller-operator, ignition-server-proxy, certified-operators-catalog, cluster-storage-operator, ignition-server, control-plane-pki-operator, olm-collect-profiles, openshift-route-controller-manager, cluster-network-operator, cluster-image-registry-operator, machine-approver, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-autoscaler, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000dc4980>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-2qbp55nw5hq6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3cab7363-1a89-4114-8a30-15cd32af7525 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3cab7363-1a89-4114-8a30-15cd32af7525", "name": "3cab7363-1a89-4114-8a30-15cd32af7525", "status": "Failed", "startTime": "2026-08-20T13:17:04.908245766Z", "endTime": "2026-08-20T13:36:10.801854971Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, konnectivity-agent, catalog-operator, packageserver, redhat-operators-catalog, community-operators-catalog, kube-controller-manager, cluster-version-operator, openshift-controller-manager, cluster-policy-controller, azure-cloud-controller-manager, olm-operator, kube-scheduler, dns-operator, ingress-operator, csi-snapshot-controller-operator, ignition-server-proxy, certified-operators-catalog, cluster-storage-operator, ignition-server, control-plane-pki-operator, olm-collect-profiles, openshift-route-controller-manager, cluster-network-operator, cluster-image-registry-operator, machine-approver, cluster-node-tuning-operator, redhat-marketplace-catalog, cluster-autoscaler, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 12:36:44 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e68800>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-rddwcsq6xrbg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce", "name": "bf0b0684-dd10-4cf3-87c3-baf48258b7ce", "status": "Failed", "startTime": "2026-08-20T13:16:38.793143062Z", "endTime": "2026-08-20T13:35:40.482871637Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e68800>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-rddwcsq6xrbg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce", "name": "bf0b0684-dd10-4cf3-87c3-baf48258b7ce", "status": "Failed", "startTime": "2026-08-20T13:16:38.793143062Z", "endTime": "2026-08-20T13:35:40.482871637Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
08:35:04	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	14	e2e (14) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc00111e108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-95b2mn6x5nc7" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425", "name": "56a5601c-84db-4c4c-afe3-fa090d79a425", "status": "Failed", "startTime": "2026-08-20T09:48:30.429013476Z", "endTime": "2026-08-20T10:12:32.646924727Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7kb8vek727d6pbio7effbv09ei0ua still exists (deletion dispatched at 2026-08-20T09:48:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc00111e108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-95b2mn6x5nc7" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425", "name": "56a5601c-84db-4c4c-afe3-fa090d79a425", "status": "Failed", "startTime": "2026-08-20T09:48:30.429013476Z", "endTime": "2026-08-20T10:12:32.646924727Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7kb8vek727d6pbio7effbv09ei0ua still exists (deletion dispatched at 2026-08-20T09:48:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc001b026c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-447nnn7lv7ss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68bcf8b9-c7f6-4e6f-9f16-24314ebf9371 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68bcf8b9-c7f6-4e6f-9f16-24314ebf9371", "name": "68bcf8b9-c7f6-4e6f-9f16-24314ebf9371", "status": "Failed", "startTime": "2026-08-20T08:54:41.378616139Z", "endTime": "2026-08-20T09:13:49.738147219Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc001b026c0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-447nnn7lv7ss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68bcf8b9-c7f6-4e6f-9f16-24314ebf9371 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68bcf8b9-c7f6-4e6f-9f16-24314ebf9371", "name": "68bcf8b9-c7f6-4e6f-9f16-24314ebf9371", "status": "Failed", "startTime": "2026-08-20T08:54:41.378616139Z", "endTime": "2026-08-20T09:13:49.738147219Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000895308>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list7lw9mf" in resourcegroup="ea-list-2lm6qlj6qrb9" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe059e7f-9879-4a59-879c-7c2d634a7dc3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe059e7f-9879-4a59-879c-7c2d634a7dc3", "name": "fe059e7f-9879-4a59-879c-7c2d634a7dc3", "status": "Failed", "startTime": "2026-08-20T09:44:55.575829675Z", "endTime": "2026-08-20T10:09:02.513171982Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7keqbo3pl38n1vug7bjfptpeikrn1 still exists (deletion dispatched at 2026-08-20T09:44:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc000895308>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list7lw9mf" in resourcegroup="ea-list-2lm6qlj6qrb9" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe059e7f-9879-4a59-879c-7c2d634a7dc3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe059e7f-9879-4a59-879c-7c2d634a7dc3", "name": "fe059e7f-9879-4a59-879c-7c2d634a7dc3", "status": "Failed", "startTime": "2026-08-20T09:44:55.575829675Z", "endTime": "2026-08-20T10:09:02.513171982Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7keqbo3pl38n1vug7bjfptpeikrn1 still exists (deletion dispatched at 2026-08-20T09:44:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000de13e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-2kbvdf4gpb7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158", "name": "1ea7be1d-979f-43d0-a6eb-4fadf5b52158", "status": "Failed", "startTime": "2026-08-20T09:49:43.253613436Z", "endTime": "2026-08-20T10:08:52.449510653Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000de13e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-2kbvdf4gpb7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158", "name": "1ea7be1d-979f-43d0-a6eb-4fadf5b52158", "status": "Failed", "startTime": "2026-08-20T09:49:43.253613436Z", "endTime": "2026-08-20T10:08:52.449510653Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc002d34258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-x729kk" in resourcegroup="rg-candidate-4-22-6wvs8lznc8fp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c", "name": "bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c", "status": "Failed", "startTime": "2026-08-20T09:47:07.957787635Z", "endTime": "2026-08-20T10:11:12.923142399Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7h5aduc2uvihvfe2vdijl20h32glr still exists (deletion dispatched at 2026-08-20T09:47:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc002d34258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-x729kk" in resourcegroup="rg-candidate-4-22-6wvs8lznc8fp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c", "name": "bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c", "status": "Failed", "startTime": "2026-08-20T09:47:07.957787635Z", "endTime": "2026-08-20T10:11:12.923142399Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7h5aduc2uvihvfe2vdijl20h32glr still exists (deletion dispatched at 2026-08-20T09:47:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-gpmtwfgcr6xn/cluster-candidate-4-20-6mrg8j should provision Unexpected error: <fmt.wrapError \| 0xc001090160>: failed to create HCP cluster cluster-candidate-4-20-6mrg8j: failed waiting for cluster="cluster-candidate-4-20-6mrg8j" in resourcegroup="rg-candidate-4-20-gpmtwfgcr6xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4", "name": "a1f59ecc-759d-4b6c-8b93-932b0208adf4", "status": "Failed", "startTime": "2026-08-20T08:53:33.518746975Z", "endTime": "2026-08-20T09:12:39.900608486Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-gpmtwfgcr6xn/cluster-candidate-4-20-6mrg8j should provision Unexpected error: <fmt.wrapError \| 0xc001090160>: failed to create HCP cluster cluster-candidate-4-20-6mrg8j: failed waiting for cluster="cluster-candidate-4-20-6mrg8j" in resourcegroup="rg-candidate-4-20-gpmtwfgcr6xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4", "name": "a1f59ecc-759d-4b6c-8b93-932b0208adf4", "status": "Failed", "startTime": "2026-08-20T08:53:33.518746975Z", "endTime": "2026-08-20T09:12:39.900608486Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-sttt5r at version 4.22.10 Unexpected error: <fmt.wrapError \| 0xc000f130e0>: failed to create HCP cluster np-skip-minor-sttt5r: failed waiting for cluster="np-skip-minor-sttt5r" in resourcegroup="rg-np-skip-minor-sttt5r-4fsvkfwk5vdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa", "name": "a570b418-a849-489b-8b7e-8e8fb8135baa", "status": "Failed", "startTime": "2026-08-20T09:36:05.972495231Z", "endTime": "2026-08-20T09:55:11.507838131Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, azure-cloud-controller-manager, dns-operator, olm-operator, machine-approver, kube-controller-manager, hosted-cluster-config-operator, kube-storage-version-migrator, catalog-operator, olm-collect-profiles, kube-scheduler, community-operators-catalog, cluster-autoscaler, openshift-route-controller-manager, cluster-policy-controller, csi-snapshot-controller-operator, packageserver, control-plane-pki-operator, cluster-storage-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, openshift-apiserver, cluster-image-registry-operator, redhat-marketplace-catalog, certified-operators-catalog, openshift-controller-manager, cluster-network-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-sttt5r at version 4.22.10 Unexpected error: <fmt.wrapError \| 0xc000f130e0>: failed to create HCP cluster np-skip-minor-sttt5r: failed waiting for cluster="np-skip-minor-sttt5r" in resourcegroup="rg-np-skip-minor-sttt5r-4fsvkfwk5vdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa", "name": "a570b418-a849-489b-8b7e-8e8fb8135baa", "status": "Failed", "startTime": "2026-08-20T09:36:05.972495231Z", "endTime": "2026-08-20T09:55:11.507838131Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, azure-cloud-controller-manager, dns-operator, olm-operator, machine-approver, kube-controller-manager, hosted-cluster-config-operator, kube-storage-version-migrator, catalog-operator, olm-collect-profiles, kube-scheduler, community-operators-catalog, cluster-autoscaler, openshift-route-controller-manager, cluster-policy-controller, csi-snapshot-controller-operator, packageserver, control-plane-pki-operator, cluster-storage-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, openshift-apiserver, cluster-image-registry-operator, redhat-marketplace-catalog, certified-operators-catalog, openshift-controller-manager, cluster-network-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ae0da0>: failed to create HCP cluster np-downgrade-wt9mk2: failed waiting for cluster="np-downgrade-wt9mk2" in resourcegroup="rg-np-downgrade-wt9mk2-ftnm4xhg5vhq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e349a722-afad-443f-add1-19a731db69f3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e349a722-afad-443f-add1-19a731db69f3", "name": "e349a722-afad-443f-add1-19a731db69f3", "status": "Failed", "startTime": "2026-08-20T09:36:41.289744282Z", "endTime": "2026-08-20T09:55:41.524756796Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ae0da0>: failed to create HCP cluster np-downgrade-wt9mk2: failed waiting for cluster="np-downgrade-wt9mk2" in resourcegroup="rg-np-downgrade-wt9mk2-ftnm4xhg5vhq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e349a722-afad-443f-add1-19a731db69f3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e349a722-afad-443f-add1-19a731db69f3", "name": "e349a722-afad-443f-add1-19a731db69f3", "status": "Failed", "startTime": "2026-08-20T09:36:41.289744282Z", "endTime": "2026-08-20T09:55:41.524756796Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-21 with version 4.21.28 Unexpected error: <fmt.wrapError \| 0xc0010bc7e0>: failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-8mw8p7" in resourcegroup="rg-np-version-upgrade-8mw8p7-jzxbk7cd5ktl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d", "name": "217c5b5b-876e-4dfd-aa27-a2f787a06b7d", "status": "Failed", "startTime": "2026-08-20T09:52:42.575458861Z", "endTime": "2026-08-20T10:11:42.477430185Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-21 with version 4.21.28 Unexpected error: <fmt.wrapError \| 0xc0010bc7e0>: failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-8mw8p7" in resourcegroup="rg-np-version-upgrade-8mw8p7-jzxbk7cd5ktl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d", "name": "217c5b5b-876e-4dfd-aa27-a2f787a06b7d", "status": "Failed", "startTime": "2026-08-20T09:52:42.575458861Z", "endTime": "2026-08-20T10:11:42.477430185Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000caeac0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-lfvfr4f2rlsg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f18bc740-6f82-415a-81e5-6d3a6a94b912 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f18bc740-6f82-415a-81e5-6d3a6a94b912", "name": "f18bc740-6f82-415a-81e5-6d3a6a94b912", "status": "Failed", "startTime": "2026-08-20T09:43:33.944124188Z", "endTime": "2026-08-20T10:02:41.325901249Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000caeac0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-lfvfr4f2rlsg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f18bc740-6f82-415a-81e5-6d3a6a94b912 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f18bc740-6f82-415a-81e5-6d3a6a94b912", "name": "f18bc740-6f82-415a-81e5-6d3a6a94b912", "status": "Failed", "startTime": "2026-08-20T09:43:33.944124188Z", "endTime": "2026-08-20T10:02:41.325901249Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, ignition-server, redhat-marketplace-catalog]; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000f7af60>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-kfjt8dvwt56j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12", "name": "54f121a6-620a-4f93-bac1-3e84405aac12", "status": "Failed", "startTime": "2026-08-20T09:44:43.862276988Z", "endTime": "2026-08-20T10:03:52.490075448Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.a8j1x1l8p8y5l8a.lecf.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000f7af60>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-kfjt8dvwt56j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12", "name": "54f121a6-620a-4f93-bac1-3e84405aac12", "status": "Failed", "startTime": "2026-08-20T09:44:43.862276988Z", "endTime": "2026-08-20T10:03:52.490075448Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.a8j1x1l8p8y5l8a.lecf.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000f6ed60>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-vp6x5vhlr7vh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0187c1e1-0a74-4fa5-a1a0-3c36f116fe55 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0187c1e1-0a74-4fa5-a1a0-3c36f116fe55", "name": "0187c1e1-0a74-4fa5-a1a0-3c36f116fe55", "status": "Failed", "startTime": "2026-08-20T09:30:45.72149794Z", "endTime": "2026-08-20T09:49:51.406315863Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ignition-server-proxy, dns-operator, ingress-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, cluster-storage-operator, openshift-controller-manager, openshift-route-controller-manager, packageserver, olm-collect-profiles, machine-approver, olm-operator, control-plane-pki-operator, openshift-apiserver, cluster-policy-controller, hosted-cluster-config-operator, redhat-operators-catalog, community-operators-catalog, catalog-operator, ignition-server, certified-operators-catalog, konnectivity-agent, cluster-network-operator, cluster-autoscaler; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000f6ed60>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-vp6x5vhlr7vh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0187c1e1-0a74-4fa5-a1a0-3c36f116fe55 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0187c1e1-0a74-4fa5-a1a0-3c36f116fe55", "name": "0187c1e1-0a74-4fa5-a1a0-3c36f116fe55", "status": "Failed", "startTime": "2026-08-20T09:30:45.72149794Z", "endTime": "2026-08-20T09:49:51.406315863Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ignition-server-proxy, dns-operator, ingress-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, cluster-storage-operator, openshift-controller-manager, openshift-route-controller-manager, packageserver, olm-collect-profiles, machine-approver, olm-operator, control-plane-pki-operator, openshift-apiserver, cluster-policy-controller, hosted-cluster-config-operator, redhat-operators-catalog, community-operators-catalog, catalog-operator, ignition-server, certified-operators-catalog, konnectivity-agent, cluster-network-operator, cluster-autoscaler; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 08:35:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001084138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-9zcvtfmqbq5k" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5a33047-0436-48bd-99bc-1757d78738bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5a33047-0436-48bd-99bc-1757d78738bf", "name": "d5a33047-0436-48bd-99bc-1757d78738bf", "status": "Failed", "startTime": "2026-08-20T09:46:19.133487927Z", "endTime": "2026-08-20T10:10:22.487349206Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7kgqae0lr73vi3huktro85eu2akrr still exists (deletion dispatched at 2026-08-20T09:46:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc001084138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-9zcvtfmqbq5k" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5a33047-0436-48bd-99bc-1757d78738bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5a33047-0436-48bd-99bc-1757d78738bf", "name": "d5a33047-0436-48bd-99bc-1757d78738bf", "status": "Failed", "startTime": "2026-08-20T09:46:19.133487927Z", "endTime": "2026-08-20T10:10:22.487349206Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7kgqae0lr73vi3huktro85eu2akrr still exists (deletion dispatched at 2026-08-20T09:46:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred x509: certificate signed by unknown authority 08:35:04 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.001s. expect ingress certificate to be signed by a trusted Azure CA Expected success, but got an error: <x509.UnknownAuthorityError>: x509: certificate signed by unknown authority ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.001s. expect ingress certificate to be signed by a trusted Azure CA Expected success, but got an error: <x509.UnknownAuthorityError>: x509: certificate signed by unknown authority ...
03:16:28	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	2	e2e (2) ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift 03:16:28 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000678260>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-lvmrprf9szxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659", "name": "dc452150-a44c-4cdf-95f9-3ab1762bc659", "status": "Failed", "startTime": "2026-08-20T03:43:58.925008531Z", "endTime": "2026-08-20T04:03:00.154367927Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000678260>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-lvmrprf9szxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659", "name": "dc452150-a44c-4cdf-95f9-3ab1762bc659", "status": "Failed", "startTime": "2026-08-20T03:43:58.925008531Z", "endTime": "2026-08-20T04:03:00.154367927Z", "error": { "code": "InternalServerError", "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift 03:16:28 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc0009eaaa0>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qjg88gwvq5xj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4", "name": "c2facd99-ace4-452a-af16-9d4629a551d4", "status": "Failed", "startTime": "2026-08-20T04:35:11.139884226Z", "endTime": "2026-08-20T04:54:12.658497794Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints Unexpected error: <fmt.wrapError \| 0xc0009eaaa0>: failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qjg88gwvq5xj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4", "name": "c2facd99-ace4-452a-af16-9d4629a551d4", "status": "Failed", "startTime": "2026-08-20T04:35:11.139884226Z", "endTime": "2026-08-20T04:54:12.658497794Z", "error": { "code": "InternalServerError", "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e" } } -------------------------------------------------------------------------------- ... occurred
02:00:07	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	5	e2e (5) ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.29; provider Microsoft.RedHatOpenShift 02:00:07 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-qkbdvr" to minor version 4.21 Unexpected error: <exported.ResponseError \| 0xc0008bcae0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qdqdx9dwcprj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-qkbdvr -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": may not decrease from 4.21.28", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be at least 4.21.29", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-qkbdvr" to minor version 4.21 Unexpected error: <exported.ResponseError \| 0xc0008bcae0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qdqdx9dwcprj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-qkbdvr -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": may not decrease from 4.21.28", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be at least 4.21.29", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 02:00:07 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc00176e318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-hkbsbzplhbxk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6deb1d14-3e14-491b-b92f-53c5344ce486 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6deb1d14-3e14-491b-b92f-53c5344ce486", "name": "6deb1d14-3e14-491b-b92f-53c5344ce486", "status": "Failed", "startTime": "2026-08-20T02:19:48.926061657Z", "endTime": "2026-08-20T02:43:48.575119816Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa13vuihbvlq5ba5p02g5kj4iq47j36 still exists (deletion dispatched at 2026-08-20T02:19:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error: <errors.joinError \| 0xc00176e318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-hkbsbzplhbxk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6deb1d14-3e14-491b-b92f-53c5344ce486 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6deb1d14-3e14-491b-b92f-53c5344ce486", "name": "6deb1d14-3e14-491b-b92f-53c5344ce486", "status": "Failed", "startTime": "2026-08-20T02:19:48.926061657Z", "endTime": "2026-08-20T02:43:48.575119816Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa13vuihbvlq5ba5p02g5kj4iq47j36 still exists (deletion dispatched at 2026-08-20T02:19:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 02:00:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-dgkn5p9l9flc/cluster-candidate-5-0-nrmbvd should provision Unexpected error: <fmt.wrapErrors \| 0xc000a85590>: failed to create HCP cluster cluster-candidate-5-0-nrmbvd, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-nrmbvd in resource group rg-candidate-5-0-dgkn5p9l9flc, error: failed waiting for cluster="cluster-candidate-5-0-nrmbvd" in resourcegroup="rg-candidate-5-0-dgkn5p9l9flc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-nrmbvd in resource group rg-candidate-5-0-dgkn5p9l9flc, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-dgkn5p9l9flc/cluster-candidate-5-0-nrmbvd should provision Unexpected error: <fmt.wrapErrors \| 0xc000a85590>: failed to create HCP cluster cluster-candidate-5-0-nrmbvd, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-nrmbvd in resource group rg-candidate-5-0-dgkn5p9l9flc, error: failed waiting for cluster="cluster-candidate-5-0-nrmbvd" in resourcegroup="rg-candidate-5-0-dgkn5p9l9flc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-nrmbvd in resource group rg-candidate-5-0-dgkn5p9l9flc, error: context deadline exceeded ... occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.34; provider Microsoft.RedHatOpenShift 02:00:07 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-4lxbwp" to minor version 4.20 Unexpected error: <exported.ResponseError \| 0xc0011b1e00>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p9w9mgdh26kn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-4lxbwp -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": may not decrease from 4.20.33", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": must be at least 4.20.34", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-4lxbwp" to minor version 4.20 Unexpected error: <exported.ResponseError \| 0xc0011b1e00>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p9w9mgdh26kn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-4lxbwp -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": may not decrease from 4.20.33", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20\": must be at least 4.20.34", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.22": must be at least 4.22.9; provider Microsoft.RedHatOpenShift 02:00:07 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-fnbgn2" to minor version 4.22 Unexpected error: <exported.ResponseError \| 0xc0011309f0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-cbhdqjqgzmjz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-fnbgn2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": may not decrease from 4.22.9", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": must be at least 4.22.9", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-fnbgn2" to minor version 4.22 Unexpected error: <exported.ResponseError \| 0xc0011309f0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-cbhdqjqgzmjz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-fnbgn2 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": may not decrease from 4.22.9", "target": "properties.version.id" }, { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": must be at least 4.22.9", "target": "properties.version.id" } ] } } -------------------------------------------------------------------------------- ... occurred