CIHealth Run Log

Environment: PROD

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
19:00:12	2h 47m 31s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	eastus2euap	e2e	failed	n/a	63	e2e (63) ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-l85jxrxmvfg7/cluster-candidate-5-1-ghbggt should provision Unexpected error: <fmt.wrapError \| 0x709092aac80>: failed to create HCP cluster cluster-candidate-5-1-ghbggt: failed waiting for cluster="cluster-candidate-5-1-ghbggt" in resourcegroup="rg-candidate-5-1-l85jxrxmvfg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac8055d4-30a4-4ff3-95b6-1f02d345f839 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac8055d4-30a4-4ff3-95b6-1f02d345f839", "name": "ac8055d4-30a4-4ff3-95b6-1f02d345f839", "status": "Failed", "startTime": "2026-10-28T20:10:42.248648198Z", "endTime": "2026-10-28T20:39:45.04100346Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac8055d4-30a4-4ff3-95b6-1f02d345f839 correlation ID: 2a70045d-9d54-4fca-878d-5f4bb11971b7 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-l85jxrxmvfg7/cluster-candidate-5-1-ghbggt should provision Unexpected error: <fmt.wrapError \| 0x709092aac80>: failed to create HCP cluster cluster-candidate-5-1-ghbggt: failed waiting for cluster="cluster-candidate-5-1-ghbggt" in resourcegroup="rg-candidate-5-1-l85jxrxmvfg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac8055d4-30a4-4ff3-95b6-1f02d345f839 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac8055d4-30a4-4ff3-95b6-1f02d345f839", "name": "ac8055d4-30a4-4ff3-95b6-1f02d345f839", "status": "Failed", "startTime": "2026-10-28T20:10:42.248648198Z", "endTime": "2026-10-28T20:39:45.04100346Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac8055d4-30a4-4ff3-95b6-1f02d345f839 correlation ID: 2a70045d-9d54-4fca-878d-5f4bb11971b7 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-s7k2sd" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x297062e2e820>: failed to create HCP cluster cp-ystream-upgrade-4-23-s7k2sd: failed waiting for cluster="cp-ystream-upgrade-4-23-s7k2sd" in resourcegroup="rg-cp-ystream-upgrade-4-23-np7xjvwc8xct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460", "name": "5d9c960a-8f4b-476d-bd5b-132c0ce76460", "status": "Failed", "startTime": "2026-10-28T20:12:32.825742556Z", "endTime": "2026-10-28T20:41:35.067951058Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460 correlation ID: 00646159-dae9-4d36-b0f6-38a33eb4750d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-s7k2sd" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x297062e2e820>: failed to create HCP cluster cp-ystream-upgrade-4-23-s7k2sd: failed waiting for cluster="cp-ystream-upgrade-4-23-s7k2sd" in resourcegroup="rg-cp-ystream-upgrade-4-23-np7xjvwc8xct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460", "name": "5d9c960a-8f4b-476d-bd5b-132c0ce76460", "status": "Failed", "startTime": "2026-10-28T20:12:32.825742556Z", "endTime": "2026-10-28T20:41:35.067951058Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460 correlation ID: 00646159-dae9-4d36-b0f6-38a33eb4750d client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-controller-manager, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0x3da3f19209c0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10", "name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10", "status": "Failed", "startTime": "2026-10-28T20:14:36.875741431Z", "endTime": "2026-10-28T20:43:44.887394978Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10 correlation ID: 84abeaa9-4051-418b-b066-32246401955a client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0x3da3f19209c0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10", "name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10", "status": "Failed", "startTime": "2026-10-28T20:14:36.875741431Z", "endTime": "2026-10-28T20:43:44.887394978Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10 correlation ID: 84abeaa9-4051-418b-b066-32246401955a client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x13016dc5e5e0>: failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d", "name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d", "status": "Failed", "startTime": "2026-10-28T20:12:34.851456324Z", "endTime": "2026-10-28T20:41:34.873934441Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x13016dc5e5e0>: failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d", "name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d", "status": "Failed", "startTime": "2026-10-28T20:12:34.851456324Z", "endTime": "2026-10-28T20:41:34.873934441Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc Unexpected error: <fmt.wrapError \| 0xd7eab0d0620>: failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b", "name": "36728157-067b-4375-a959-9c9ce43e534b", "status": "Failed", "startTime": "2026-10-28T20:56:26.269135944Z", "endTime": "2026-10-28T21:25:30.687402107Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc Unexpected error: <fmt.wrapError \| 0xd7eab0d0620>: failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b", "name": "36728157-067b-4375-a959-9c9ce43e534b", "status": "Failed", "startTime": "2026-10-28T20:56:26.269135944Z", "endTime": "2026-10-28T21:25:30.687402107Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision Unexpected error: <fmt.wrapError \| 0x15a8256fc4a0>: failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238", "name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238", "status": "Failed", "startTime": "2026-10-28T20:04:15.708912895Z", "endTime": "2026-10-28T20:33:16.216619801Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238 correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision Unexpected error: <fmt.wrapError \| 0x15a8256fc4a0>: failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238", "name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238", "status": "Failed", "startTime": "2026-10-28T20:04:15.708912895Z", "endTime": "2026-10-28T20:33:16.216619801Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238 correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0x34cbd3d68900>: failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6", "name": "91a3c445-9466-4d25-8cd8-de90f532d7c6", "status": "Failed", "startTime": "2026-10-28T20:14:32.731497539Z", "endTime": "2026-10-28T20:43:34.905565303Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6 correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0x34cbd3d68900>: failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6", "name": "91a3c445-9466-4d25-8cd8-de90f532d7c6", "status": "Failed", "startTime": "2026-10-28T20:14:32.731497539Z", "endTime": "2026-10-28T20:43:34.905565303Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6 correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xd149dec0ae0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d", "name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d", "status": "Failed", "startTime": "2026-10-28T19:24:20.936404311Z", "endTime": "2026-10-28T19:53:23.678300779Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xd149dec0ae0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d", "name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d", "status": "Failed", "startTime": "2026-10-28T19:24:20.936404311Z", "endTime": "2026-10-28T19:53:23.678300779Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0x2190c42a2260>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-nd5rwvspn9b8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71207653-6aed-4ccc-a2bd-10690ebc31ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71207653-6aed-4ccc-a2bd-10690ebc31ba", "name": "71207653-6aed-4ccc-a2bd-10690ebc31ba", "status": "Failed", "startTime": "2026-10-28T20:49:04.122773503Z", "endTime": "2026-10-28T21:18:10.583197461Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, packageserver, certified-operators-catalog, konnectivity-agent, cluster-policy-controller, cluster-node-tuning-operator, openshift-controller-manager, cluster-storage-operator, cluster-network-operator, dns-operator, cluster-image-registry-operator, olm-operator, community-operators-catalog, openshift-route-controller-manager, redhat-operators-catalog, redhat-marketplace-catalog, kube-scheduler, ingress-operator, csi-snapshot-controller-operator, catalog-operator, cluster-version-operator, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71207653-6aed-4ccc-a2bd-10690ebc31ba correlation ID: dfe646a0-0df6-424d-ac15-19436441fd56 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0x2190c42a2260>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-nd5rwvspn9b8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71207653-6aed-4ccc-a2bd-10690ebc31ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71207653-6aed-4ccc-a2bd-10690ebc31ba", "name": "71207653-6aed-4ccc-a2bd-10690ebc31ba", "status": "Failed", "startTime": "2026-10-28T20:49:04.122773503Z", "endTime": "2026-10-28T21:18:10.583197461Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, packageserver, certified-operators-catalog, konnectivity-agent, cluster-policy-controller, cluster-node-tuning-operator, openshift-controller-manager, cluster-storage-operator, cluster-network-operator, dns-operator, cluster-image-registry-operator, olm-operator, community-operators-catalog, openshift-route-controller-manager, redhat-operators-catalog, redhat-marketplace-catalog, kube-scheduler, ingress-operator, csi-snapshot-controller-operator, catalog-operator, cluster-version-operator, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71207653-6aed-4ccc-a2bd-10690ebc31ba correlation ID: dfe646a0-0df6-424d-ac15-19436441fd56 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-g2ccqb at version 4.22.20 Unexpected error: <fmt.wrapError \| 0x3d077ee36720>: failed to create HCP cluster np-skip-minor-g2ccqb: failed waiting for cluster="np-skip-minor-g2ccqb" in resourcegroup="rg-np-skip-minor-g2ccqb-mn87dp5nj29s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1799e75d-c12f-41af-9c14-45990959eb21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1799e75d-c12f-41af-9c14-45990959eb21", "name": "1799e75d-c12f-41af-9c14-45990959eb21", "status": "Failed", "startTime": "2026-10-28T20:57:08.441802556Z", "endTime": "2026-10-28T21:26:10.61116359Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, packageserver, hosted-cluster-config-operator, cluster-network-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, cluster-policy-controller, konnectivity-agent, cluster-storage-operator, openshift-controller-manager, ingress-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, dns-operator, kube-storage-version-migrator, catalog-operator, community-operators-catalog, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1799e75d-c12f-41af-9c14-45990959eb21 correlation ID: 79286a08-358d-49d2-868d-a28ffe41cd8c client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-g2ccqb at version 4.22.20 Unexpected error: <fmt.wrapError \| 0x3d077ee36720>: failed to create HCP cluster np-skip-minor-g2ccqb: failed waiting for cluster="np-skip-minor-g2ccqb" in resourcegroup="rg-np-skip-minor-g2ccqb-mn87dp5nj29s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1799e75d-c12f-41af-9c14-45990959eb21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1799e75d-c12f-41af-9c14-45990959eb21", "name": "1799e75d-c12f-41af-9c14-45990959eb21", "status": "Failed", "startTime": "2026-10-28T20:57:08.441802556Z", "endTime": "2026-10-28T21:26:10.61116359Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, packageserver, hosted-cluster-config-operator, cluster-network-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, cluster-policy-controller, konnectivity-agent, cluster-storage-operator, openshift-controller-manager, ingress-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, dns-operator, kube-storage-version-migrator, catalog-operator, community-operators-catalog, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1799e75d-c12f-41af-9c14-45990959eb21 correlation ID: 79286a08-358d-49d2-868d-a28ffe41cd8c client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-g7572t" with version 4.21.39 on candidate channel Unexpected error: <fmt.wrapError \| 0x5e791d44820>: failed to create HCP cluster cluster-zstream-4-21-g7572t: failed waiting for cluster="cluster-zstream-4-21-g7572t" in resourcegroup="rg-zstream-upgrade-4-21-9k9lb5xrt68d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd19c6e4-5a58-48c7-894c-aa0a8d05d58c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd19c6e4-5a58-48c7-894c-aa0a8d05d58c", "name": "cd19c6e4-5a58-48c7-894c-aa0a8d05d58c", "status": "Failed", "startTime": "2026-10-28T20:10:53.702291365Z", "endTime": "2026-10-28T20:39:55.046264536Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, catalog-operator, redhat-operators-catalog, redhat-marketplace-catalog, machine-approver, kube-scheduler, cluster-autoscaler, openshift-apiserver, azure-cloud-controller-manager, community-operators-catalog, openshift-controller-manager, ingress-operator, control-plane-pki-operator, cluster-network-operator, packageserver, olm-collect-profiles, certified-operators-catalog, cluster-policy-controller, cluster-storage-operator, cluster-image-registry-operator, olm-operator, kube-controller-manager, cluster-version-operator, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd19c6e4-5a58-48c7-894c-aa0a8d05d58c correlation ID: 75d43cba-42bf-4725-aaf6-4b8f69707172 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-g7572t" with version 4.21.39 on candidate channel Unexpected error: <fmt.wrapError \| 0x5e791d44820>: failed to create HCP cluster cluster-zstream-4-21-g7572t: failed waiting for cluster="cluster-zstream-4-21-g7572t" in resourcegroup="rg-zstream-upgrade-4-21-9k9lb5xrt68d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd19c6e4-5a58-48c7-894c-aa0a8d05d58c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd19c6e4-5a58-48c7-894c-aa0a8d05d58c", "name": "cd19c6e4-5a58-48c7-894c-aa0a8d05d58c", "status": "Failed", "startTime": "2026-10-28T20:10:53.702291365Z", "endTime": "2026-10-28T20:39:55.046264536Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, catalog-operator, redhat-operators-catalog, redhat-marketplace-catalog, machine-approver, kube-scheduler, cluster-autoscaler, openshift-apiserver, azure-cloud-controller-manager, community-operators-catalog, openshift-controller-manager, ingress-operator, control-plane-pki-operator, cluster-network-operator, packageserver, olm-collect-profiles, certified-operators-catalog, cluster-policy-controller, cluster-storage-operator, cluster-image-registry-operator, olm-operator, kube-controller-manager, cluster-version-operator, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cd19c6e4-5a58-48c7-894c-aa0a8d05d58c correlation ID: 75d43cba-42bf-4725-aaf6-4b8f69707172 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0x1683f5097100>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b", "name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b", "status": "Failed", "startTime": "2026-10-28T19:28:18.023653922Z", "endTime": "2026-10-28T19:57:23.841118908Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0x1683f5097100>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b", "name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b", "status": "Failed", "startTime": "2026-10-28T19:28:18.023653922Z", "endTime": "2026-10-28T19:57:23.841118908Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x14a89c71a940>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tpg6t2mn29xj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/35be8b72-7d8c-49bd-8ea2-93aca817ec52 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/35be8b72-7d8c-49bd-8ea2-93aca817ec52", "name": "35be8b72-7d8c-49bd-8ea2-93aca817ec52", "status": "Failed", "startTime": "2026-10-28T19:24:19.423180703Z", "endTime": "2026-10-28T19:53:23.854104536Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/35be8b72-7d8c-49bd-8ea2-93aca817ec52 correlation ID: 7e7906b1-9aa5-44db-80a6-483bb9566fac client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x14a89c71a940>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tpg6t2mn29xj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/35be8b72-7d8c-49bd-8ea2-93aca817ec52 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/35be8b72-7d8c-49bd-8ea2-93aca817ec52", "name": "35be8b72-7d8c-49bd-8ea2-93aca817ec52", "status": "Failed", "startTime": "2026-10-28T19:24:19.423180703Z", "endTime": "2026-10-28T19:53:23.854104536Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/35be8b72-7d8c-49bd-8ea2-93aca817ec52 correlation ID: 7e7906b1-9aa5-44db-80a6-483bb9566fac client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision Unexpected error: <fmt.wrapError \| 0xba6b52aa80>: failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd", "name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd", "status": "Failed", "startTime": "2026-10-28T20:10:20.02531103Z", "endTime": "2026-10-28T20:39:25.022579343Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd correlation ID: 15324628-570b-42b9-80a7-dbd435939211 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision Unexpected error: <fmt.wrapError \| 0xba6b52aa80>: failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd", "name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd", "status": "Failed", "startTime": "2026-10-28T20:10:20.02531103Z", "endTime": "2026-10-28T20:39:25.022579343Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd correlation ID: 15324628-570b-42b9-80a7-dbd435939211 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision Unexpected error: <fmt.wrapError \| 0x3e94b832e820>: failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5", "name": "e7fb8359-1343-4d42-8975-4e247aae3bc5", "status": "Failed", "startTime": "2026-10-28T20:06:06.856819198Z", "endTime": "2026-10-28T20:35:15.000751357Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5 correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision Unexpected error: <fmt.wrapError \| 0x3e94b832e820>: failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5", "name": "e7fb8359-1343-4d42-8975-4e247aae3bc5", "status": "Failed", "startTime": "2026-10-28T20:06:06.856819198Z", "endTime": "2026-10-28T20:35:15.000751357Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5 correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0x2c03b21d6c20>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26", "name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26", "status": "Failed", "startTime": "2026-10-28T20:14:37.658468995Z", "endTime": "2026-10-28T20:43:44.89685741Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26 correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0x2c03b21d6c20>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26", "name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26", "status": "Failed", "startTime": "2026-10-28T20:14:37.658468995Z", "endTime": "2026-10-28T20:43:44.89685741Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26 correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-xkwqdc" with version 4.23.0-ec.0 on candidate channel Unexpected error: <fmt.wrapError \| 0x3dceb2f8ab20>: failed to create HCP cluster cluster-zstream-4-23-xkwqdc: failed waiting for cluster="cluster-zstream-4-23-xkwqdc" in resourcegroup="rg-zstream-upgrade-4-23-blql6slsnrzk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0020dfe1-eb87-4673-bb31-be80a802433f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0020dfe1-eb87-4673-bb31-be80a802433f", "name": "0020dfe1-eb87-4673-bb31-be80a802433f", "status": "Failed", "startTime": "2026-10-28T20:11:05.420114587Z", "endTime": "2026-10-28T20:40:15.034303339Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, dns-operator, konnectivity-agent, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, hosted-cluster-config-operator, openshift-controller-manager, cluster-storage-operator, ingress-operator, packageserver, community-operators-catalog, kube-controller-manager, cluster-image-registry-operator, csi-snapshot-controller-operator, olm-operator, certified-operators-catalog, cluster-policy-controller, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0020dfe1-eb87-4673-bb31-be80a802433f correlation ID: cc2d8ce8-0861-418e-90fe-22edf335d211 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-xkwqdc" with version 4.23.0-ec.0 on candidate channel Unexpected error: <fmt.wrapError \| 0x3dceb2f8ab20>: failed to create HCP cluster cluster-zstream-4-23-xkwqdc: failed waiting for cluster="cluster-zstream-4-23-xkwqdc" in resourcegroup="rg-zstream-upgrade-4-23-blql6slsnrzk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0020dfe1-eb87-4673-bb31-be80a802433f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0020dfe1-eb87-4673-bb31-be80a802433f", "name": "0020dfe1-eb87-4673-bb31-be80a802433f", "status": "Failed", "startTime": "2026-10-28T20:11:05.420114587Z", "endTime": "2026-10-28T20:40:15.034303339Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, dns-operator, konnectivity-agent, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, hosted-cluster-config-operator, openshift-controller-manager, cluster-storage-operator, ingress-operator, packageserver, community-operators-catalog, kube-controller-manager, cluster-image-registry-operator, csi-snapshot-controller-operator, olm-operator, certified-operators-catalog, cluster-policy-controller, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0020dfe1-eb87-4673-bb31-be80a802433f correlation ID: cc2d8ce8-0861-418e-90fe-22edf335d211 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-crm2px" with version 4.20.43 on candidate channel Unexpected error: <fmt.wrapError \| 0x5d990898640>: failed to create HCP cluster cluster-zstream-4-20-crm2px: failed waiting for cluster="cluster-zstream-4-20-crm2px" in resourcegroup="rg-zstream-upgrade-4-20-ktl4cbqz8hgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/227090e2-0a2d-418d-b6f5-f3e92ddc87ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/227090e2-0a2d-418d-b6f5-f3e92ddc87ca", "name": "227090e2-0a2d-418d-b6f5-f3e92ddc87ca", "status": "Failed", "startTime": "2026-10-28T20:11:00.297311043Z", "endTime": "2026-10-28T20:40:05.017131119Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-marketplace-catalog, certified-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-policy-controller, catalog-operator, packageserver, olm-collect-profiles, openshift-controller-manager, azure-cloud-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, redhat-operators-catalog, capi-provider, control-plane-pki-operator, cluster-version-operator, cluster-image-registry-operator, konnectivity-agent, olm-operator, kube-controller-manager, openshift-apiserver, openshift-route-controller-manager, community-operators-catalog, cluster-autoscaler, cluster-network-operator, dns-operator, ingress-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/227090e2-0a2d-418d-b6f5-f3e92ddc87ca correlation ID: a0e92328-63d3-4953-90ed-c8b08db12589 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-crm2px" with version 4.20.43 on candidate channel Unexpected error: <fmt.wrapError \| 0x5d990898640>: failed to create HCP cluster cluster-zstream-4-20-crm2px: failed waiting for cluster="cluster-zstream-4-20-crm2px" in resourcegroup="rg-zstream-upgrade-4-20-ktl4cbqz8hgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/227090e2-0a2d-418d-b6f5-f3e92ddc87ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/227090e2-0a2d-418d-b6f5-f3e92ddc87ca", "name": "227090e2-0a2d-418d-b6f5-f3e92ddc87ca", "status": "Failed", "startTime": "2026-10-28T20:11:00.297311043Z", "endTime": "2026-10-28T20:40:05.017131119Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-marketplace-catalog, certified-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-policy-controller, catalog-operator, packageserver, olm-collect-profiles, openshift-controller-manager, azure-cloud-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, redhat-operators-catalog, capi-provider, control-plane-pki-operator, cluster-version-operator, cluster-image-registry-operator, konnectivity-agent, olm-operator, kube-controller-manager, openshift-apiserver, openshift-route-controller-manager, community-operators-catalog, cluster-autoscaler, cluster-network-operator, dns-operator, ingress-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/227090e2-0a2d-418d-b6f5-f3e92ddc87ca correlation ID: a0e92328-63d3-4953-90ed-c8b08db12589 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0x2a9a9c302780>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b", "name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b", "status": "Failed", "startTime": "2026-10-28T20:14:46.976794807Z", "endTime": "2026-10-28T20:43:55.053369978Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0x2a9a9c302780>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b", "name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b", "status": "Failed", "startTime": "2026-10-28T20:14:46.976794807Z", "endTime": "2026-10-28T20:43:55.053369978Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-lrsdfl" at install version 4.20 Unexpected error: <fmt.wrapError \| 0x3c350f225e0>: failed to create HCP cluster cp-ystream-upgrade-4-21-lrsdfl: failed waiting for cluster="cp-ystream-upgrade-4-21-lrsdfl" in resourcegroup="rg-cp-ystream-upgrade-4-21-vfbxfhgg5xwh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616", "name": "034eed87-a977-47cf-8ff5-2b8292aa4616", "status": "Failed", "startTime": "2026-10-28T20:12:34.208698264Z", "endTime": "2026-10-28T20:41:35.072616907Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616 correlation ID: 880b9f59-ff57-4ef3-9ec1-2561f0663b62 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-lrsdfl" at install version 4.20 Unexpected error: <fmt.wrapError \| 0x3c350f225e0>: failed to create HCP cluster cp-ystream-upgrade-4-21-lrsdfl: failed waiting for cluster="cp-ystream-upgrade-4-21-lrsdfl" in resourcegroup="rg-cp-ystream-upgrade-4-21-vfbxfhgg5xwh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616", "name": "034eed87-a977-47cf-8ff5-2b8292aa4616", "status": "Failed", "startTime": "2026-10-28T20:12:34.208698264Z", "endTime": "2026-10-28T20:41:35.072616907Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616 correlation ID: 880b9f59-ff57-4ef3-9ec1-2561f0663b62 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0x118c82e68960>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f", "name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f", "status": "Failed", "startTime": "2026-10-28T20:58:03.268943167Z", "endTime": "2026-10-28T21:27:10.630246232Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0x118c82e68960>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f", "name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f", "status": "Failed", "startTime": "2026-10-28T20:58:03.268943167Z", "endTime": "2026-10-28T21:27:10.630246232Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0x22868c3a2660>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53", "name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53", "status": "Failed", "startTime": "2026-10-28T19:24:09.588021132Z", "endTime": "2026-10-28T19:53:13.758028769Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53 correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0x22868c3a2660>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53", "name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53", "status": "Failed", "startTime": "2026-10-28T19:24:09.588021132Z", "endTime": "2026-10-28T19:53:13.758028769Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53 correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71 client request ID: unavailable occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 19:00:12 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0xfa629f24ea0>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xfa629c70d20>{ s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0xfa629f24ea0>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xfa629c70d20>{ s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0x1308b97a65a0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96", "name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96", "status": "Failed", "startTime": "2026-10-28T20:58:09.127282806Z", "endTime": "2026-10-28T21:27:10.682711777Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96 correlation ID: 49b88924-064f-4f52-9480-109cf607a390 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0x1308b97a65a0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96", "name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96", "status": "Failed", "startTime": "2026-10-28T20:58:09.127282806Z", "endTime": "2026-10-28T21:27:10.682711777Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96 correlation ID: 49b88924-064f-4f52-9480-109cf607a390 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0x2840a3490800>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e", "name": "f493db72-5a08-4ef3-8c73-4d055bd2007e", "status": "Failed", "startTime": "2026-10-28T20:48:29.562853635Z", "endTime": "2026-10-28T21:17:30.601011159Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0x2840a3490800>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e", "name": "f493db72-5a08-4ef3-8c73-4d055bd2007e", "status": "Failed", "startTime": "2026-10-28T20:48:29.562853635Z", "endTime": "2026-10-28T21:17:30.601011159Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0x23a60cd6e680>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc", "name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc", "status": "Failed", "startTime": "2026-10-28T19:24:25.094565507Z", "endTime": "2026-10-28T19:53:33.692195544Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0x23a60cd6e680>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc", "name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc", "status": "Failed", "startTime": "2026-10-28T19:24:25.094565507Z", "endTime": "2026-10-28T19:53:33.692195544Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, openshift-controller-manager, openshift-route-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0x1d947782ea20>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e", "name": "0a9f3022-f355-43e2-9259-81dfd53aa74e", "status": "Failed", "startTime": "2026-10-28T20:01:08.393167298Z", "endTime": "2026-10-28T20:30:16.146752142Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0x1d947782ea20>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e", "name": "0a9f3022-f355-43e2-9259-81dfd53aa74e", "status": "Failed", "startTime": "2026-10-28T20:01:08.393167298Z", "endTime": "2026-10-28T20:30:16.146752142Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x3d00e53ba4e0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227", "name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227", "status": "Failed", "startTime": "2026-10-28T19:24:17.131948207Z", "endTime": "2026-10-28T19:53:23.815769706Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227 correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x3d00e53ba4e0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227", "name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227", "status": "Failed", "startTime": "2026-10-28T19:24:17.131948207Z", "endTime": "2026-10-28T19:53:23.815769706Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227 correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70 client request ID: unavailable occurred Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 19:00:12 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x1cf3d70a0be0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-vjtmsffknq7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44acf888-443c-4577-88c8-87d955fdc85d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44acf888-443c-4577-88c8-87d955fdc85d", "name": "44acf888-443c-4577-88c8-87d955fdc85d", "status": "Failed", "startTime": "2026-10-28T19:24:30.864672464Z", "endTime": "2026-10-28T19:53:33.757605836Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.1 is Partial (want Completed), started 24m49s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44acf888-443c-4577-88c8-87d955fdc85d correlation ID: e7629f4b-1e00-4dc0-b99d-16a5be46a759 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x1cf3d70a0be0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-vjtmsffknq7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44acf888-443c-4577-88c8-87d955fdc85d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44acf888-443c-4577-88c8-87d955fdc85d", "name": "44acf888-443c-4577-88c8-87d955fdc85d", "status": "Failed", "startTime": "2026-10-28T19:24:30.864672464Z", "endTime": "2026-10-28T19:53:33.757605836Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.1 is Partial (want Completed), started 24m49s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44acf888-443c-4577-88c8-87d955fdc85d correlation ID: e7629f4b-1e00-4dc0-b99d-16a5be46a759 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0x2df91d94ebc0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a", "name": "57f0f2bf-9284-42db-8d6e-b13db18de25a", "status": "Failed", "startTime": "2026-10-28T19:52:14.429975606Z", "endTime": "2026-10-28T20:21:16.149275515Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0x2df91d94ebc0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a", "name": "57f0f2bf-9284-42db-8d6e-b13db18de25a", "status": "Failed", "startTime": "2026-10-28T19:52:14.429975606Z", "endTime": "2026-10-28T20:21:16.149275515Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40 Unexpected error: <fmt.wrapError \| 0x9790da22600>: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e", "name": "e487a831-bc1a-4554-9930-0ef5e5dd164e", "status": "Failed", "startTime": "2026-10-28T20:55:25.009646036Z", "endTime": "2026-10-28T21:24:30.625209621Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40 Unexpected error: <fmt.wrapError \| 0x9790da22600>: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e", "name": "e487a831-bc1a-4554-9930-0ef5e5dd164e", "status": "Failed", "startTime": "2026-10-28T20:55:25.009646036Z", "endTime": "2026-10-28T21:24:30.625209621Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc983e72b20>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1", "name": "338180fd-fd37-4118-8ce5-da19d007d2d1", "status": "Failed", "startTime": "2026-10-28T19:24:20.13733405Z", "endTime": "2026-10-28T19:53:23.880336461Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1 correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc983e72b20>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1", "name": "338180fd-fd37-4118-8ce5-da19d007d2d1", "status": "Failed", "startTime": "2026-10-28T19:24:20.13733405Z", "endTime": "2026-10-28T19:53:23.880336461Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1 correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0x14a5e2008300>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466", "name": "a961a70a-5b5a-4175-bb58-82fb36beb466", "status": "Failed", "startTime": "2026-10-28T19:26:12.743400806Z", "endTime": "2026-10-28T19:55:13.7057976Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466 correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0x14a5e2008300>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466", "name": "a961a70a-5b5a-4175-bb58-82fb36beb466", "status": "Failed", "startTime": "2026-10-28T19:26:12.743400806Z", "endTime": "2026-10-28T19:55:13.7057976Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466 correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001 Unexpected error: <fmt.wrapError \| 0x110402dd07a0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-l4d74x5hc8gz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6987ce55-b0fd-4dcd-9644-01b84a500116 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6987ce55-b0fd-4dcd-9644-01b84a500116", "name": "6987ce55-b0fd-4dcd-9644-01b84a500116", "status": "Failed", "startTime": "2026-10-28T19:27:48.339569357Z", "endTime": "2026-10-28T19:56:53.862034082Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6987ce55-b0fd-4dcd-9644-01b84a500116 correlation ID: 324dd373-c202-4e02-bc01-3d2aa84e7912 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001 Unexpected error: <fmt.wrapError \| 0x110402dd07a0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-l4d74x5hc8gz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6987ce55-b0fd-4dcd-9644-01b84a500116 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6987ce55-b0fd-4dcd-9644-01b84a500116", "name": "6987ce55-b0fd-4dcd-9644-01b84a500116", "status": "Failed", "startTime": "2026-10-28T19:27:48.339569357Z", "endTime": "2026-10-28T19:56:53.862034082Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6987ce55-b0fd-4dcd-9644-01b84a500116 correlation ID: 324dd373-c202-4e02-bc01-3d2aa84e7912 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x1da2d78427a0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0", "name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0", "status": "Failed", "startTime": "2026-10-28T19:26:05.081696809Z", "endTime": "2026-10-28T19:55:13.864538233Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0 correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x1da2d78427a0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0", "name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0", "status": "Failed", "startTime": "2026-10-28T19:26:05.081696809Z", "endTime": "2026-10-28T19:55:13.864538233Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0 correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision Unexpected error: <fmt.wrapError \| 0x104ef3fa7e0>: failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b", "name": "4e3df969-37c8-42e0-bc9c-a513f08b762b", "status": "Failed", "startTime": "2026-10-28T20:10:48.049950576Z", "endTime": "2026-10-28T20:39:55.034999436Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision Unexpected error: <fmt.wrapError \| 0x104ef3fa7e0>: failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b", "name": "4e3df969-37c8-42e0-bc9c-a513f08b762b", "status": "Failed", "startTime": "2026-10-28T20:10:48.049950576Z", "endTime": "2026-10-28T20:39:55.034999436Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0x3b3d00e688d0>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0", "name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0", "status": "Failed", "startTime": "2026-10-28T20:59:31.361013478Z", "endTime": "2026-10-28T21:28:40.621642045Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0 correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0x3b3d00e688d0>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0", "name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0", "status": "Failed", "startTime": "2026-10-28T20:59:31.361013478Z", "endTime": "2026-10-28T21:28:40.621642045Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0 correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-sgk9ch" at install version 4.21 Unexpected error: <fmt.wrapError \| 0x12b89f7eb40>: failed to create HCP cluster cp-ystream-upgrade-4-22-sgk9ch: failed waiting for cluster="cp-ystream-upgrade-4-22-sgk9ch" in resourcegroup="rg-cp-ystream-upgrade-4-22-27jpzmw77lts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/14601d65-c9c2-4ca1-aaf0-80f2a475267f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/14601d65-c9c2-4ca1-aaf0-80f2a475267f", "name": "14601d65-c9c2-4ca1-aaf0-80f2a475267f", "status": "Failed", "startTime": "2026-10-28T20:12:33.485207001Z", "endTime": "2026-10-28T20:41:35.087630634Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, packageserver, redhat-operators-catalog, kube-scheduler, olm-operator, kube-controller-manager, cluster-storage-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog, ingress-operator, csi-snapshot-controller-operator, cluster-version-operator, dns-operator, openshift-route-controller-manager, cluster-network-operator, konnectivity-agent, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/14601d65-c9c2-4ca1-aaf0-80f2a475267f correlation ID: 0f2608cc-c139-4188-9e46-cb83d44b4b49 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-sgk9ch" at install version 4.21 Unexpected error: <fmt.wrapError \| 0x12b89f7eb40>: failed to create HCP cluster cp-ystream-upgrade-4-22-sgk9ch: failed waiting for cluster="cp-ystream-upgrade-4-22-sgk9ch" in resourcegroup="rg-cp-ystream-upgrade-4-22-27jpzmw77lts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/14601d65-c9c2-4ca1-aaf0-80f2a475267f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/14601d65-c9c2-4ca1-aaf0-80f2a475267f", "name": "14601d65-c9c2-4ca1-aaf0-80f2a475267f", "status": "Failed", "startTime": "2026-10-28T20:12:33.485207001Z", "endTime": "2026-10-28T20:41:35.087630634Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, packageserver, redhat-operators-catalog, kube-scheduler, olm-operator, kube-controller-manager, cluster-storage-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog, ingress-operator, csi-snapshot-controller-operator, cluster-version-operator, dns-operator, openshift-route-controller-manager, cluster-network-operator, konnectivity-agent, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/14601d65-c9c2-4ca1-aaf0-80f2a475267f correlation ID: 0f2608cc-c139-4188-9e46-cb83d44b4b49 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40 Unexpected error: <fmt.wrapError \| 0x1c6ab1a4a4c0>: failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996", "name": "13f26b26-2324-48a4-82df-d8b324b13996", "status": "Failed", "startTime": "2026-10-28T20:52:21.783931077Z", "endTime": "2026-10-28T21:21:30.723052246Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996 correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40 Unexpected error: <fmt.wrapError \| 0x1c6ab1a4a4c0>: failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996", "name": "13f26b26-2324-48a4-82df-d8b324b13996", "status": "Failed", "startTime": "2026-10-28T20:52:21.783931077Z", "endTime": "2026-10-28T21:21:30.723052246Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996 correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x34142edd6a40>: failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8", "name": "f063aff6-fa6b-4397-9b14-f08322d800e8", "status": "Failed", "startTime": "2026-10-28T19:24:14.723008935Z", "endTime": "2026-10-28T19:53:23.809094912Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8 correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x34142edd6a40>: failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8", "name": "f063aff6-fa6b-4397-9b14-f08322d800e8", "status": "Failed", "startTime": "2026-10-28T19:24:14.723008935Z", "endTime": "2026-10-28T19:53:23.809094912Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8 correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xe23c97da540>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3", "name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3", "status": "Failed", "startTime": "2026-10-28T20:28:51.316103778Z", "endTime": "2026-10-28T20:57:55.004654509Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3 correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xe23c97da540>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3", "name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3", "status": "Failed", "startTime": "2026-10-28T20:28:51.316103778Z", "endTime": "2026-10-28T20:57:55.004654509Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3 correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x38e5d805a900>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-w6g9pxqkxlb5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f038d0c-188e-4fd1-b9b6-4a92361ba19c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f038d0c-188e-4fd1-b9b6-4a92361ba19c", "name": "8f038d0c-188e-4fd1-b9b6-4a92361ba19c", "status": "Failed", "startTime": "2026-10-28T19:24:20.372217471Z", "endTime": "2026-10-28T19:53:23.8370741Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f038d0c-188e-4fd1-b9b6-4a92361ba19c correlation ID: 634adffc-fd16-4c0c-b46c-854f988d728f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x38e5d805a900>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-w6g9pxqkxlb5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f038d0c-188e-4fd1-b9b6-4a92361ba19c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f038d0c-188e-4fd1-b9b6-4a92361ba19c", "name": "8f038d0c-188e-4fd1-b9b6-4a92361ba19c", "status": "Failed", "startTime": "2026-10-28T19:24:20.372217471Z", "endTime": "2026-10-28T19:53:23.8370741Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8f038d0c-188e-4fd1-b9b6-4a92361ba19c correlation ID: 634adffc-fd16-4c0c-b46c-854f988d728f client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0x1d67385a4720>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf", "name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf", "status": "Failed", "startTime": "2026-10-28T19:24:18.787179838Z", "endTime": "2026-10-28T19:53:23.770490111Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0x1d67385a4720>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf", "name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf", "status": "Failed", "startTime": "2026-10-28T19:24:18.787179838Z", "endTime": "2026-10-28T19:53:23.770490111Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x364bf26c2280>: failed to create HCP cluster np-dg-minor-82f95j: failed waiting for cluster="np-dg-minor-82f95j" in resourcegroup="rg-np-dg-minor-82f95j-47wd4n6jxmd6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4c41c18d-b7df-4635-a0c8-e62246a9f100 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4c41c18d-b7df-4635-a0c8-e62246a9f100", "name": "4c41c18d-b7df-4635-a0c8-e62246a9f100", "status": "Failed", "startTime": "2026-10-28T20:57:20.485792808Z", "endTime": "2026-10-28T21:26:20.677519522Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-policy-controller, hosted-cluster-config-operator, dns-operator, certified-operators-catalog, machine-approver, control-plane-pki-operator, cluster-network-operator, olm-operator, redhat-operators-catalog, azure-cloud-controller-manager, cluster-image-registry-operator, catalog-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, packageserver, cluster-version-operator, csi-snapshot-controller-operator, kube-storage-version-migrator, kube-scheduler, cluster-storage-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4c41c18d-b7df-4635-a0c8-e62246a9f100 correlation ID: 68c0efa5-5e54-499c-9838-0520a414503e client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x364bf26c2280>: failed to create HCP cluster np-dg-minor-82f95j: failed waiting for cluster="np-dg-minor-82f95j" in resourcegroup="rg-np-dg-minor-82f95j-47wd4n6jxmd6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4c41c18d-b7df-4635-a0c8-e62246a9f100 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4c41c18d-b7df-4635-a0c8-e62246a9f100", "name": "4c41c18d-b7df-4635-a0c8-e62246a9f100", "status": "Failed", "startTime": "2026-10-28T20:57:20.485792808Z", "endTime": "2026-10-28T21:26:20.677519522Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-policy-controller, hosted-cluster-config-operator, dns-operator, certified-operators-catalog, machine-approver, control-plane-pki-operator, cluster-network-operator, olm-operator, redhat-operators-catalog, azure-cloud-controller-manager, cluster-image-registry-operator, catalog-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, packageserver, cluster-version-operator, csi-snapshot-controller-operator, kube-storage-version-migrator, kube-scheduler, cluster-storage-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4c41c18d-b7df-4635-a0c8-e62246a9f100 correlation ID: 68c0efa5-5e54-499c-9838-0520a414503e client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0x10164f31ca00>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p669mb2thnqj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b", "name": "f26dc7e7-f380-4b37-87cb-db720fa2be9b", "status": "Failed", "startTime": "2026-10-28T19:27:58.648367476Z", "endTime": "2026-10-28T19:57:03.816857754Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b correlation ID: b3870728-f2f7-48de-a0fa-7f5442586990 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0x10164f31ca00>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p669mb2thnqj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b", "name": "f26dc7e7-f380-4b37-87cb-db720fa2be9b", "status": "Failed", "startTime": "2026-10-28T19:27:58.648367476Z", "endTime": "2026-10-28T19:57:03.816857754Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b correlation ID: b3870728-f2f7-48de-a0fa-7f5442586990 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x16967f5def20>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749", "name": "54478daa-47e5-4530-91c2-342cc29a1749", "status": "Failed", "startTime": "2026-10-28T19:24:21.628677591Z", "endTime": "2026-10-28T19:53:23.838328082Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749 correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x16967f5def20>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749", "name": "54478daa-47e5-4530-91c2-342cc29a1749", "status": "Failed", "startTime": "2026-10-28T19:24:21.628677591Z", "endTime": "2026-10-28T19:53:23.838328082Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749 correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0x28ad82ebac20>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117", "name": "d2708466-dcce-4112-be7d-6e0e6a3a5117", "status": "Failed", "startTime": "2026-10-28T20:01:11.67931549Z", "endTime": "2026-10-28T20:30:16.178438369Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117 correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0x28ad82ebac20>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117", "name": "d2708466-dcce-4112-be7d-6e0e6a3a5117", "status": "Failed", "startTime": "2026-10-28T20:01:11.67931549Z", "endTime": "2026-10-28T20:30:16.178438369Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117 correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-t46hvnqkdqvf/cluster-candidate-4-22-2nwg57 should provision Unexpected error: <fmt.wrapError \| 0xa071317e240>: failed to create HCP cluster cluster-candidate-4-22-2nwg57: failed waiting for cluster="cluster-candidate-4-22-2nwg57" in resourcegroup="rg-candidate-4-22-t46hvnqkdqvf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa78952a-2a0e-4c91-9b40-6d20c521b765 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa78952a-2a0e-4c91-9b40-6d20c521b765", "name": "fa78952a-2a0e-4c91-9b40-6d20c521b765", "status": "Failed", "startTime": "2026-10-28T20:10:11.743652526Z", "endTime": "2026-10-28T20:39:15.054194298Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa78952a-2a0e-4c91-9b40-6d20c521b765 correlation ID: 3428fa79-40a9-4a5a-8766-7a9690285732 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-t46hvnqkdqvf/cluster-candidate-4-22-2nwg57 should provision Unexpected error: <fmt.wrapError \| 0xa071317e240>: failed to create HCP cluster cluster-candidate-4-22-2nwg57: failed waiting for cluster="cluster-candidate-4-22-2nwg57" in resourcegroup="rg-candidate-4-22-t46hvnqkdqvf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa78952a-2a0e-4c91-9b40-6d20c521b765 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa78952a-2a0e-4c91-9b40-6d20c521b765", "name": "fa78952a-2a0e-4c91-9b40-6d20c521b765", "status": "Failed", "startTime": "2026-10-28T20:10:11.743652526Z", "endTime": "2026-10-28T20:39:15.054194298Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa78952a-2a0e-4c91-9b40-6d20c521b765 correlation ID: 3428fa79-40a9-4a5a-8766-7a9690285732 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0x38cab388f240>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-z6f4tl45wbl6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62ac4023-4fb6-42a6-87c4-4a745d49d80b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62ac4023-4fb6-42a6-87c4-4a745d49d80b", "name": "62ac4023-4fb6-42a6-87c4-4a745d49d80b", "status": "Failed", "startTime": "2026-10-28T19:28:41.52216937Z", "endTime": "2026-10-28T19:57:43.694575383Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62ac4023-4fb6-42a6-87c4-4a745d49d80b correlation ID: 0bc7323c-15c6-4cfa-85ba-27c1c70f38cc client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0x38cab388f240>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-z6f4tl45wbl6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62ac4023-4fb6-42a6-87c4-4a745d49d80b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62ac4023-4fb6-42a6-87c4-4a745d49d80b", "name": "62ac4023-4fb6-42a6-87c4-4a745d49d80b", "status": "Failed", "startTime": "2026-10-28T19:28:41.52216937Z", "endTime": "2026-10-28T19:57:43.694575383Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/62ac4023-4fb6-42a6-87c4-4a745d49d80b correlation ID: 0bc7323c-15c6-4cfa-85ba-27c1c70f38cc client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0 Unexpected error: <fmt.wrapError \| 0x2d29037be7e0>: failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4", "name": "75f18780-46df-4ca4-b758-e34d2262ebd4", "status": "Failed", "startTime": "2026-10-28T20:12:38.563887566Z", "endTime": "2026-10-28T20:41:44.890635353Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4 correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0 Unexpected error: <fmt.wrapError \| 0x2d29037be7e0>: failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4", "name": "75f18780-46df-4ca4-b758-e34d2262ebd4", "status": "Failed", "startTime": "2026-10-28T20:12:38.563887566Z", "endTime": "2026-10-28T20:41:44.890635353Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4 correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel Unexpected error: <fmt.wrapError \| 0x3981bcd4a5c0>: failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae", "name": "fa758444-9da4-4261-b65a-fb9af0416aae", "status": "Failed", "startTime": "2026-10-28T20:11:05.048147318Z", "endTime": "2026-10-28T20:40:15.036753709Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel Unexpected error: <fmt.wrapError \| 0x3981bcd4a5c0>: failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae", "name": "fa758444-9da4-4261-b65a-fb9af0416aae", "status": "Failed", "startTime": "2026-10-28T20:11:05.048147318Z", "endTime": "2026-10-28T20:40:15.036753709Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44 Unexpected error: <fmt.wrapError \| 0x1c7b6c0a8440>: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf", "name": "6a1a42fe-bb72-488b-adbf-55a400b476cf", "status": "Failed", "startTime": "2026-10-28T20:56:27.61784978Z", "endTime": "2026-10-28T21:25:30.606619249Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44 Unexpected error: <fmt.wrapError \| 0x1c7b6c0a8440>: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf", "name": "6a1a42fe-bb72-488b-adbf-55a400b476cf", "status": "Failed", "startTime": "2026-10-28T20:56:27.61784978Z", "endTime": "2026-10-28T21:25:30.606619249Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0x385465998780>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-m8h7nbml5knf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c42a163-6c2b-42e8-a29b-4e6883ba3f36 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c42a163-6c2b-42e8-a29b-4e6883ba3f36", "name": "7c42a163-6c2b-42e8-a29b-4e6883ba3f36", "status": "Failed", "startTime": "2026-10-28T19:26:03.613135527Z", "endTime": "2026-10-28T19:55:03.867198108Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c42a163-6c2b-42e8-a29b-4e6883ba3f36 correlation ID: 85b18aae-16d5-46e6-a585-e2682c6d10ca client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0x385465998780>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-m8h7nbml5knf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c42a163-6c2b-42e8-a29b-4e6883ba3f36 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c42a163-6c2b-42e8-a29b-4e6883ba3f36", "name": "7c42a163-6c2b-42e8-a29b-4e6883ba3f36", "status": "Failed", "startTime": "2026-10-28T19:26:03.613135527Z", "endTime": "2026-10-28T19:55:03.867198108Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7c42a163-6c2b-42e8-a29b-4e6883ba3f36 correlation ID: 85b18aae-16d5-46e6-a585-e2682c6d10ca client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0x3290e061a680>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9", "name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9", "status": "Failed", "startTime": "2026-10-28T20:57:27.436526224Z", "endTime": "2026-10-28T21:26:30.622804028Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9 correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0x3290e061a680>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9", "name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9", "status": "Failed", "startTime": "2026-10-28T20:57:27.436526224Z", "endTime": "2026-10-28T21:26:30.622804028Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9 correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x26ceb862cd00>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wz7pqrvxjxjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94aab46e-a3d9-462d-bb43-0c5fc752748f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94aab46e-a3d9-462d-bb43-0c5fc752748f", "name": "94aab46e-a3d9-462d-bb43-0c5fc752748f", "status": "Failed", "startTime": "2026-10-28T19:24:20.25232077Z", "endTime": "2026-10-28T19:53:23.685636564Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94aab46e-a3d9-462d-bb43-0c5fc752748f correlation ID: 745d7020-48a3-43c5-8d75-45376466cb1f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x26ceb862cd00>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wz7pqrvxjxjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94aab46e-a3d9-462d-bb43-0c5fc752748f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94aab46e-a3d9-462d-bb43-0c5fc752748f", "name": "94aab46e-a3d9-462d-bb43-0c5fc752748f", "status": "Failed", "startTime": "2026-10-28T19:24:20.25232077Z", "endTime": "2026-10-28T19:53:23.685636564Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94aab46e-a3d9-462d-bb43-0c5fc752748f correlation ID: 745d7020-48a3-43c5-8d75-45376466cb1f client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x3735652cc6a0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-nxtm6wfxc8r7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38392dc4-2811-4cda-8591-7ffe1ffac7c1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38392dc4-2811-4cda-8591-7ffe1ffac7c1", "name": "38392dc4-2811-4cda-8591-7ffe1ffac7c1", "status": "Failed", "startTime": "2026-10-28T19:24:21.68302357Z", "endTime": "2026-10-28T19:53:23.756141507Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, konnectivity-agent, control-plane-pki-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, openshift-controller-manager, ingress-operator, kube-scheduler, cluster-version-operator, csi-snapshot-controller-operator, kube-storage-version-migrator, certified-operators-catalog, kube-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, openshift-route-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, packageserver, community-operators-catalog, catalog-operator, machine-approver, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38392dc4-2811-4cda-8591-7ffe1ffac7c1 correlation ID: b11a943b-2f1f-4773-a692-76ceee856dc1 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x3735652cc6a0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-nxtm6wfxc8r7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38392dc4-2811-4cda-8591-7ffe1ffac7c1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38392dc4-2811-4cda-8591-7ffe1ffac7c1", "name": "38392dc4-2811-4cda-8591-7ffe1ffac7c1", "status": "Failed", "startTime": "2026-10-28T19:24:21.68302357Z", "endTime": "2026-10-28T19:53:23.756141507Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, konnectivity-agent, control-plane-pki-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, openshift-controller-manager, ingress-operator, kube-scheduler, cluster-version-operator, csi-snapshot-controller-operator, kube-storage-version-migrator, certified-operators-catalog, kube-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, openshift-route-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, packageserver, community-operators-catalog, catalog-operator, machine-approver, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/38392dc4-2811-4cda-8591-7ffe1ffac7c1 correlation ID: b11a943b-2f1f-4773-a692-76ceee856dc1 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0x44f1a4fe8a0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jf8m6zdj6vs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3346d2af-f75c-40f6-8165-8b6d194d6773 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3346d2af-f75c-40f6-8165-8b6d194d6773", "name": "3346d2af-f75c-40f6-8165-8b6d194d6773", "status": "Failed", "startTime": "2026-10-28T19:24:20.343807929Z", "endTime": "2026-10-28T19:53:23.794639518Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3346d2af-f75c-40f6-8165-8b6d194d6773 correlation ID: 1abc5bd8-a67b-46a9-affe-3b9e66e6ca79 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0x44f1a4fe8a0>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jf8m6zdj6vs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3346d2af-f75c-40f6-8165-8b6d194d6773 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3346d2af-f75c-40f6-8165-8b6d194d6773", "name": "3346d2af-f75c-40f6-8165-8b6d194d6773", "status": "Failed", "startTime": "2026-10-28T19:24:20.343807929Z", "endTime": "2026-10-28T19:53:23.794639518Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3346d2af-f75c-40f6-8165-8b6d194d6773 correlation ID: 1abc5bd8-a67b-46a9-affe-3b9e66e6ca79 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0x38a796b1a740>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93", "name": "499291aa-b279-429d-80ba-7786fc6cfa93", "status": "Failed", "startTime": "2026-10-28T20:38:17.873619905Z", "endTime": "2026-10-28T21:07:24.99192406Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93 correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0x38a796b1a740>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93", "name": "499291aa-b279-429d-80ba-7786fc6cfa93", "status": "Failed", "startTime": "2026-10-28T20:38:17.873619905Z", "endTime": "2026-10-28T21:07:24.99192406Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93 correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0x263f7a1ecb40>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e", "name": "bc13db95-dad4-4bc2-9929-827244e95d2e", "status": "Failed", "startTime": "2026-10-28T20:14:34.589492538Z", "endTime": "2026-10-28T20:43:34.88292258Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e correlation ID: da1e3131-8ea4-44cd-87a3-857957002869 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0x263f7a1ecb40>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e", "name": "bc13db95-dad4-4bc2-9929-827244e95d2e", "status": "Failed", "startTime": "2026-10-28T20:14:34.589492538Z", "endTime": "2026-10-28T20:43:34.88292258Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e correlation ID: da1e3131-8ea4-44cd-87a3-857957002869 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, ignition-server-proxy, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x1c9f11d305c0>: failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644", "name": "c3f52426-50e3-477e-b6d2-730dd7ef4644", "status": "Failed", "startTime": "2026-10-28T20:57:10.002650956Z", "endTime": "2026-10-28T21:26:10.63929862Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644 correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x1c9f11d305c0>: failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644", "name": "c3f52426-50e3-477e-b6d2-730dd7ef4644", "status": "Failed", "startTime": "2026-10-28T20:57:10.002650956Z", "endTime": "2026-10-28T21:26:10.63929862Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644 correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel Unexpected error: <fmt.wrapError \| 0x219ad233aca0>: failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091", "name": "e768a2f1-bf75-4afe-9acf-741914b73091", "status": "Failed", "startTime": "2026-10-28T20:11:31.970768227Z", "endTime": "2026-10-28T20:40:35.054714998Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091 correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel Unexpected error: <fmt.wrapError \| 0x219ad233aca0>: failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091", "name": "e768a2f1-bf75-4afe-9acf-741914b73091", "status": "Failed", "startTime": "2026-10-28T20:11:31.970768227Z", "endTime": "2026-10-28T20:40:35.054714998Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091 correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 19:00:12 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0x1eab57386d00>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-pq4dgm8h8d5m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1b5b8c7b-0882-4b28-b36d-8d86306ba636 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1b5b8c7b-0882-4b28-b36d-8d86306ba636", "name": "1b5b8c7b-0882-4b28-b36d-8d86306ba636", "status": "Failed", "startTime": "2026-10-28T19:24:21.125042883Z", "endTime": "2026-10-28T19:53:23.713000323Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1b5b8c7b-0882-4b28-b36d-8d86306ba636 correlation ID: 3c010b6b-955b-427d-a23b-ecd8944c4077 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0x1eab57386d00>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-pq4dgm8h8d5m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1b5b8c7b-0882-4b28-b36d-8d86306ba636 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1b5b8c7b-0882-4b28-b36d-8d86306ba636", "name": "1b5b8c7b-0882-4b28-b36d-8d86306ba636", "status": "Failed", "startTime": "2026-10-28T19:24:21.125042883Z", "endTime": "2026-10-28T19:53:23.713000323Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1b5b8c7b-0882-4b28-b36d-8d86306ba636 correlation ID: 3c010b6b-955b-427d-a23b-ecd8944c4077 client request ID: unavailable occurred
18:29:14	2h 6m 24s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	n/a	passed	n/a	0	n/a
13:09:17	3h 12m 4s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	eastus2euap	e2e	failed	n/a	66	e2e (66) ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0x1f34eadb6e20>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dbk6jgfrnzkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eacff0a3-4bd4-4c15-aafe-008fc8887810 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eacff0a3-4bd4-4c15-aafe-008fc8887810", "name": "eacff0a3-4bd4-4c15-aafe-008fc8887810", "status": "Failed", "startTime": "2026-10-28T15:21:17.380797935Z", "endTime": "2026-10-28T15:50:17.704324747Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eacff0a3-4bd4-4c15-aafe-008fc8887810 correlation ID: 3ef3ff1c-e195-4831-adde-5137de419b11 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0x1f34eadb6e20>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dbk6jgfrnzkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eacff0a3-4bd4-4c15-aafe-008fc8887810 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eacff0a3-4bd4-4c15-aafe-008fc8887810", "name": "eacff0a3-4bd4-4c15-aafe-008fc8887810", "status": "Failed", "startTime": "2026-10-28T15:21:17.380797935Z", "endTime": "2026-10-28T15:50:17.704324747Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/eacff0a3-4bd4-4c15-aafe-008fc8887810 correlation ID: 3ef3ff1c-e195-4831-adde-5137de419b11 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0x3b896839fa60>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-fld8q7cxfnnl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592", "name": "fad0e7c8-51f4-402f-b2c0-d5c286ac7592", "status": "Failed", "startTime": "2026-10-28T13:44:11.609067289Z", "endTime": "2026-10-28T14:13:20.968792662Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592 correlation ID: 19e863b6-3287-4026-a228-cda73ba74c13 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0x3b896839fa60>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-fld8q7cxfnnl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592", "name": "fad0e7c8-51f4-402f-b2c0-d5c286ac7592", "status": "Failed", "startTime": "2026-10-28T13:44:11.609067289Z", "endTime": "2026-10-28T14:13:20.968792662Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592 correlation ID: 19e863b6-3287-4026-a228-cda73ba74c13 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel Unexpected error: <fmt.wrapError \| 0x3f72bfdcc60>: failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366", "name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366", "status": "Failed", "startTime": "2026-10-28T14:39:56.97734875Z", "endTime": "2026-10-28T15:08:58.578863234Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366 correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel Unexpected error: <fmt.wrapError \| 0x3f72bfdcc60>: failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366", "name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366", "status": "Failed", "startTime": "2026-10-28T14:39:56.97734875Z", "endTime": "2026-10-28T15:08:58.578863234Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366 correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x167c557a6660>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wjkxs2684hd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d", "name": "9a50ba0b-b907-416a-b2d1-94636b24ec6d", "status": "Failed", "startTime": "2026-10-28T13:42:47.959137424Z", "endTime": "2026-10-28T14:11:50.925878227Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d correlation ID: 9f148a4c-d977-4e05-985d-74dfc49e11a0 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x167c557a6660>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wjkxs2684hd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d", "name": "9a50ba0b-b907-416a-b2d1-94636b24ec6d", "status": "Failed", "startTime": "2026-10-28T13:42:47.959137424Z", "endTime": "2026-10-28T14:11:50.925878227Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d correlation ID: 9f148a4c-d977-4e05-985d-74dfc49e11a0 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0x19f9bb4550e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jwx565kbs7q4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859613d5-8f1e-4a8a-b5fd-b35307b04c99 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859613d5-8f1e-4a8a-b5fd-b35307b04c99", "name": "859613d5-8f1e-4a8a-b5fd-b35307b04c99", "status": "Failed", "startTime": "2026-10-28T15:21:06.85183749Z", "endTime": "2026-10-28T15:50:07.785672722Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-policy-controller, ingress-operator, control-plane-pki-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, machine-approver, cluster-image-registry-operator, olm-operator, redhat-marketplace-catalog, openshift-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, community-operators-catalog, kube-controller-manager, catalog-operator, kube-scheduler, cluster-version-operator, hosted-cluster-config-operator, dns-operator, azure-cloud-controller-manager, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859613d5-8f1e-4a8a-b5fd-b35307b04c99 correlation ID: 0727a39d-6d6c-4b6d-9eb9-78345449ac83 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0x19f9bb4550e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jwx565kbs7q4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859613d5-8f1e-4a8a-b5fd-b35307b04c99 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859613d5-8f1e-4a8a-b5fd-b35307b04c99", "name": "859613d5-8f1e-4a8a-b5fd-b35307b04c99", "status": "Failed", "startTime": "2026-10-28T15:21:06.85183749Z", "endTime": "2026-10-28T15:50:07.785672722Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-policy-controller, ingress-operator, control-plane-pki-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, machine-approver, cluster-image-registry-operator, olm-operator, redhat-marketplace-catalog, openshift-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, community-operators-catalog, kube-controller-manager, catalog-operator, kube-scheduler, cluster-version-operator, hosted-cluster-config-operator, dns-operator, azure-cloud-controller-manager, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/859613d5-8f1e-4a8a-b5fd-b35307b04c99 correlation ID: 0727a39d-6d6c-4b6d-9eb9-78345449ac83 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-jj2xlg" at install version 4.21 Unexpected error: <fmt.wrapError \| 0x1496482ec2a0>: failed to create HCP cluster cp-ystream-upgrade-4-22-jj2xlg: failed waiting for cluster="cp-ystream-upgrade-4-22-jj2xlg" in resourcegroup="rg-cp-ystream-upgrade-4-22-gp9nmrbz6s6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39fc987d-482c-4621-a697-4e7a3438eafa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39fc987d-482c-4621-a697-4e7a3438eafa", "name": "39fc987d-482c-4621-a697-4e7a3438eafa", "status": "Failed", "startTime": "2026-10-28T14:40:45.234461665Z", "endTime": "2026-10-28T15:09:48.451075236Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, packageserver, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, catalog-operator, olm-operator, certified-operators-catalog, cluster-version-operator, redhat-operators-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, openshift-controller-manager, ingress-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39fc987d-482c-4621-a697-4e7a3438eafa correlation ID: b0c553ac-6da6-4f3f-b3d3-f91a300d8bd6 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-jj2xlg" at install version 4.21 Unexpected error: <fmt.wrapError \| 0x1496482ec2a0>: failed to create HCP cluster cp-ystream-upgrade-4-22-jj2xlg: failed waiting for cluster="cp-ystream-upgrade-4-22-jj2xlg" in resourcegroup="rg-cp-ystream-upgrade-4-22-gp9nmrbz6s6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39fc987d-482c-4621-a697-4e7a3438eafa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39fc987d-482c-4621-a697-4e7a3438eafa", "name": "39fc987d-482c-4621-a697-4e7a3438eafa", "status": "Failed", "startTime": "2026-10-28T14:40:45.234461665Z", "endTime": "2026-10-28T15:09:48.451075236Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, packageserver, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, catalog-operator, olm-operator, certified-operators-catalog, cluster-version-operator, redhat-operators-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, openshift-controller-manager, ingress-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/39fc987d-482c-4621-a697-4e7a3438eafa correlation ID: b0c553ac-6da6-4f3f-b3d3-f91a300d8bd6 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x1d7ea8232700>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-xz8d5m7w5nl6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88c4fef2-0e08-45c2-8eb6-f41b487cd618 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88c4fef2-0e08-45c2-8eb6-f41b487cd618", "name": "88c4fef2-0e08-45c2-8eb6-f41b487cd618", "status": "Failed", "startTime": "2026-10-28T13:44:49.504870916Z", "endTime": "2026-10-28T14:13:50.792209417Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88c4fef2-0e08-45c2-8eb6-f41b487cd618 correlation ID: 945c0bd0-a046-43be-9723-689e4959653f client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x1d7ea8232700>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-xz8d5m7w5nl6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88c4fef2-0e08-45c2-8eb6-f41b487cd618 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88c4fef2-0e08-45c2-8eb6-f41b487cd618", "name": "88c4fef2-0e08-45c2-8eb6-f41b487cd618", "status": "Failed", "startTime": "2026-10-28T13:44:49.504870916Z", "endTime": "2026-10-28T14:13:50.792209417Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/88c4fef2-0e08-45c2-8eb6-f41b487cd618 correlation ID: 945c0bd0-a046-43be-9723-689e4959653f client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0x3c6dd746e940>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-22sbv67qz5xz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8d92ce31-2582-4a90-ac50-186535026c4b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8d92ce31-2582-4a90-ac50-186535026c4b", "name": "8d92ce31-2582-4a90-ac50-186535026c4b", "status": "Failed", "startTime": "2026-10-28T15:21:56.490101151Z", "endTime": "2026-10-28T15:50:57.635126569Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, cluster-version-operator, ingress-operator, catalog-operator, redhat-marketplace-catalog, openshift-route-controller-manager, csi-snapshot-controller-operator, redhat-operators-catalog, cluster-policy-controller, dns-operator, packageserver, community-operators-catalog, cluster-network-operator, cluster-image-registry-operator, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8d92ce31-2582-4a90-ac50-186535026c4b correlation ID: b90608ee-eef4-4f50-8721-a501ceb1dc3b client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0x3c6dd746e940>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-22sbv67qz5xz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8d92ce31-2582-4a90-ac50-186535026c4b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8d92ce31-2582-4a90-ac50-186535026c4b", "name": "8d92ce31-2582-4a90-ac50-186535026c4b", "status": "Failed", "startTime": "2026-10-28T15:21:56.490101151Z", "endTime": "2026-10-28T15:50:57.635126569Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, cluster-version-operator, ingress-operator, catalog-operator, redhat-marketplace-catalog, openshift-route-controller-manager, csi-snapshot-controller-operator, redhat-operators-catalog, cluster-policy-controller, dns-operator, packageserver, community-operators-catalog, cluster-network-operator, cluster-image-registry-operator, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8d92ce31-2582-4a90-ac50-186535026c4b correlation ID: b90608ee-eef4-4f50-8721-a501ceb1dc3b client request ID: unavailable occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error: <errors.joinError \| 0x144b1f0220a8>: resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2", "name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2", "status": "Failed", "startTime": "2026-10-28T14:05:17.896484451Z", "endTime": "2026-10-28T14:29:22.39155595Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2 correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error: <errors.joinError \| 0x144b1f0220a8>: resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2", "name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2", "status": "Failed", "startTime": "2026-10-28T14:05:17.896484451Z", "endTime": "2026-10-28T14:29:22.39155595Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2 correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-21-6v7gsgmssgvq/cluster-candidate-4-21-rh49gm should provision Unexpected error: <fmt.wrapError \| 0x1da958962480>: failed to create HCP cluster cluster-candidate-4-21-rh49gm: failed waiting for cluster="cluster-candidate-4-21-rh49gm" in resourcegroup="rg-candidate-4-21-6v7gsgmssgvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606", "name": "8ae5c7d2-bbb4-424b-8ab6-07626d56a606", "status": "Failed", "startTime": "2026-10-28T14:39:59.612142564Z", "endTime": "2026-10-28T15:09:08.641921276Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606 correlation ID: 8f132f7f-c309-4e23-a969-712eed0e7d91 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-21-6v7gsgmssgvq/cluster-candidate-4-21-rh49gm should provision Unexpected error: <fmt.wrapError \| 0x1da958962480>: failed to create HCP cluster cluster-candidate-4-21-rh49gm: failed waiting for cluster="cluster-candidate-4-21-rh49gm" in resourcegroup="rg-candidate-4-21-6v7gsgmssgvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606", "name": "8ae5c7d2-bbb4-424b-8ab6-07626d56a606", "status": "Failed", "startTime": "2026-10-28T14:39:59.612142564Z", "endTime": "2026-10-28T15:09:08.641921276Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606 correlation ID: 8f132f7f-c309-4e23-a969-712eed0e7d91 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-20-5lpmjgvvtgz9/cluster-candidate-4-20-2tt2j5 should provision Unexpected error: <fmt.wrapError \| 0x1a4de4ffaa20>: failed to create HCP cluster cluster-candidate-4-20-2tt2j5: failed waiting for cluster="cluster-candidate-4-20-2tt2j5" in resourcegroup="rg-candidate-4-20-5lpmjgvvtgz9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be77f741-8538-41d7-b3a5-893532fe7a72 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be77f741-8538-41d7-b3a5-893532fe7a72", "name": "be77f741-8538-41d7-b3a5-893532fe7a72", "status": "Failed", "startTime": "2026-10-28T14:39:20.152945198Z", "endTime": "2026-10-28T15:08:28.627516087Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be77f741-8538-41d7-b3a5-893532fe7a72 correlation ID: 5f3ab66f-143a-4347-ba87-f5cc1c238cc2 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-20-5lpmjgvvtgz9/cluster-candidate-4-20-2tt2j5 should provision Unexpected error: <fmt.wrapError \| 0x1a4de4ffaa20>: failed to create HCP cluster cluster-candidate-4-20-2tt2j5: failed waiting for cluster="cluster-candidate-4-20-2tt2j5" in resourcegroup="rg-candidate-4-20-5lpmjgvvtgz9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be77f741-8538-41d7-b3a5-893532fe7a72 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be77f741-8538-41d7-b3a5-893532fe7a72", "name": "be77f741-8538-41d7-b3a5-893532fe7a72", "status": "Failed", "startTime": "2026-10-28T14:39:20.152945198Z", "endTime": "2026-10-28T15:08:28.627516087Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be77f741-8538-41d7-b3a5-893532fe7a72 correlation ID: 5f3ab66f-143a-4347-ba87-f5cc1c238cc2 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-23-x4xcrxkpwn7j/cluster-candidate-4-23-vkx5ps should provision Unexpected error: <fmt.wrapError \| 0x8d9b5b08c80>: failed to create HCP cluster cluster-candidate-4-23-vkx5ps: failed waiting for cluster="cluster-candidate-4-23-vkx5ps" in resourcegroup="rg-candidate-4-23-x4xcrxkpwn7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/195b0201-f716-4d56-87ea-883b4e6144d4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/195b0201-f716-4d56-87ea-883b4e6144d4", "name": "195b0201-f716-4d56-87ea-883b4e6144d4", "status": "Failed", "startTime": "2026-10-28T14:39:48.141950924Z", "endTime": "2026-10-28T15:08:48.609704416Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/195b0201-f716-4d56-87ea-883b4e6144d4 correlation ID: c7513d16-8aa2-4392-b9e7-c38f94dba9a2 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-23-x4xcrxkpwn7j/cluster-candidate-4-23-vkx5ps should provision Unexpected error: <fmt.wrapError \| 0x8d9b5b08c80>: failed to create HCP cluster cluster-candidate-4-23-vkx5ps: failed waiting for cluster="cluster-candidate-4-23-vkx5ps" in resourcegroup="rg-candidate-4-23-x4xcrxkpwn7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/195b0201-f716-4d56-87ea-883b4e6144d4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/195b0201-f716-4d56-87ea-883b4e6144d4", "name": "195b0201-f716-4d56-87ea-883b4e6144d4", "status": "Failed", "startTime": "2026-10-28T14:39:48.141950924Z", "endTime": "2026-10-28T15:08:48.609704416Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/195b0201-f716-4d56-87ea-883b4e6144d4 correlation ID: c7513d16-8aa2-4392-b9e7-c38f94dba9a2 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0x374e833daa40>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-f4hlsn5n7dp5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44", "name": "1f756a31-37f0-40e0-82d6-7323816b3a44", "status": "Failed", "startTime": "2026-10-28T13:42:59.191596418Z", "endTime": "2026-10-28T14:12:00.890716747Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44 correlation ID: 35fe35ef-43a8-4363-855c-128586bb4a72 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0x374e833daa40>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-f4hlsn5n7dp5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44", "name": "1f756a31-37f0-40e0-82d6-7323816b3a44", "status": "Failed", "startTime": "2026-10-28T13:42:59.191596418Z", "endTime": "2026-10-28T14:12:00.890716747Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44 correlation ID: 35fe35ef-43a8-4363-855c-128586bb4a72 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-wbzs2x at version 4.22.20 Unexpected error: <fmt.wrapError \| 0x105f86c0c5c0>: failed to create HCP cluster np-skip-minor-wbzs2x: failed waiting for cluster="np-skip-minor-wbzs2x" in resourcegroup="rg-np-skip-minor-wbzs2x-9657t9v84pkg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4", "name": "4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4", "status": "Failed", "startTime": "2026-10-28T15:21:40.618908142Z", "endTime": "2026-10-28T15:50:47.631173688Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, azure-cloud-controller-manager, machine-approver, control-plane-pki-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, cluster-storage-operator, dns-operator, cluster-image-registry-operator, kube-storage-version-migrator, catalog-operator, olm-operator, openshift-controller-manager, cluster-policy-controller, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4 correlation ID: 0acb7fcf-366a-4480-a74c-2c2b50fa0aa0 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-wbzs2x at version 4.22.20 Unexpected error: <fmt.wrapError \| 0x105f86c0c5c0>: failed to create HCP cluster np-skip-minor-wbzs2x: failed waiting for cluster="np-skip-minor-wbzs2x" in resourcegroup="rg-np-skip-minor-wbzs2x-9657t9v84pkg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4", "name": "4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4", "status": "Failed", "startTime": "2026-10-28T15:21:40.618908142Z", "endTime": "2026-10-28T15:50:47.631173688Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, azure-cloud-controller-manager, machine-approver, control-plane-pki-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, cluster-storage-operator, dns-operator, cluster-image-registry-operator, kube-storage-version-migrator, catalog-operator, olm-operator, openshift-controller-manager, cluster-policy-controller, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4 correlation ID: 0acb7fcf-366a-4480-a74c-2c2b50fa0aa0 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xdfd4e022a0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qlszp5g8k9wc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e", "name": "06deff3f-928f-4528-ab39-5d28dd6fa99e", "status": "Failed", "startTime": "2026-10-28T14:41:22.27269296Z", "endTime": "2026-10-28T15:10:28.476284696Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, openshift-controller-manager, cluster-network-operator, dns-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-policy-controller, certified-operators-catalog, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-version-operator, ingress-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e correlation ID: 09ac3f0b-4c54-44f8-a6e3-2ded2abcc35b client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xdfd4e022a0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qlszp5g8k9wc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e", "name": "06deff3f-928f-4528-ab39-5d28dd6fa99e", "status": "Failed", "startTime": "2026-10-28T14:41:22.27269296Z", "endTime": "2026-10-28T15:10:28.476284696Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, openshift-controller-manager, cluster-network-operator, dns-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-policy-controller, certified-operators-catalog, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-version-operator, ingress-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e correlation ID: 09ac3f0b-4c54-44f8-a6e3-2ded2abcc35b client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-ffl2k6" with version 4.20.43 on candidate channel Unexpected error: <fmt.wrapError \| 0x13122735c680>: failed to create HCP cluster cluster-zstream-4-20-ffl2k6: failed waiting for cluster="cluster-zstream-4-20-ffl2k6" in resourcegroup="rg-zstream-upgrade-4-20-2qwm89mjrzcr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c", "name": "ab266e95-32c5-4fd4-a60d-96a8a32d075c", "status": "Failed", "startTime": "2026-10-28T14:40:33.890129033Z", "endTime": "2026-10-28T15:09:38.467450003Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-collect-profiles, redhat-operators-catalog, machine-approver, kube-controller-manager, hosted-cluster-config-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, cluster-storage-operator, redhat-marketplace-catalog, openshift-apiserver, cluster-version-operator, catalog-operator, kube-scheduler, openshift-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, cluster-autoscaler, certified-operators-catalog, azure-cloud-controller-manager, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c correlation ID: ba9ee4d9-fd17-4e45-845d-93a236933c1e client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-ffl2k6" with version 4.20.43 on candidate channel Unexpected error: <fmt.wrapError \| 0x13122735c680>: failed to create HCP cluster cluster-zstream-4-20-ffl2k6: failed waiting for cluster="cluster-zstream-4-20-ffl2k6" in resourcegroup="rg-zstream-upgrade-4-20-2qwm89mjrzcr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c", "name": "ab266e95-32c5-4fd4-a60d-96a8a32d075c", "status": "Failed", "startTime": "2026-10-28T14:40:33.890129033Z", "endTime": "2026-10-28T15:09:38.467450003Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-collect-profiles, redhat-operators-catalog, machine-approver, kube-controller-manager, hosted-cluster-config-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, cluster-storage-operator, redhat-marketplace-catalog, openshift-apiserver, cluster-version-operator, catalog-operator, kube-scheduler, openshift-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, cluster-autoscaler, certified-operators-catalog, azure-cloud-controller-manager, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c correlation ID: ba9ee4d9-fd17-4e45-845d-93a236933c1e client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x3e45224b0c80>: failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-flztpcd4f9nh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/56a19993-1d4b-4c82-bc4c-b9cb17083165 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/56a19993-1d4b-4c82-bc4c-b9cb17083165", "name": "56a19993-1d4b-4c82-bc4c-b9cb17083165", "status": "Failed", "startTime": "2026-10-28T13:43:33.142193131Z", "endTime": "2026-10-28T14:12:40.962875758Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/56a19993-1d4b-4c82-bc4c-b9cb17083165 correlation ID: 6856df15-8a41-4e59-8005-8b021a531d3b client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x3e45224b0c80>: failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-flztpcd4f9nh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/56a19993-1d4b-4c82-bc4c-b9cb17083165 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/56a19993-1d4b-4c82-bc4c-b9cb17083165", "name": "56a19993-1d4b-4c82-bc4c-b9cb17083165", "status": "Failed", "startTime": "2026-10-28T13:43:33.142193131Z", "endTime": "2026-10-28T14:12:40.962875758Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/56a19993-1d4b-4c82-bc4c-b9cb17083165 correlation ID: 6856df15-8a41-4e59-8005-8b021a531d3b client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf Unexpected error: <fmt.wrapError \| 0x257516f84460>: failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc", "name": "92128bbf-b05b-4a42-b590-261b4b996ffc", "status": "Failed", "startTime": "2026-10-28T15:21:31.746407548Z", "endTime": "2026-10-28T15:50:37.764425706Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf Unexpected error: <fmt.wrapError \| 0x257516f84460>: failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc", "name": "92128bbf-b05b-4a42-b590-261b4b996ffc", "status": "Failed", "startTime": "2026-10-28T15:21:31.746407548Z", "endTime": "2026-10-28T15:50:37.764425706Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-8fz8f5" with version 4.21.39 on candidate channel Unexpected error: <fmt.wrapError \| 0x17985ea62800>: failed to create HCP cluster cluster-zstream-4-21-8fz8f5: failed waiting for cluster="cluster-zstream-4-21-8fz8f5" in resourcegroup="rg-zstream-upgrade-4-21-wbqzpvwdc6q6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c", "name": "487512e6-43f7-42b5-a694-f4be2585d80c", "status": "Failed", "startTime": "2026-10-28T14:40:20.778612196Z", "endTime": "2026-10-28T15:09:28.776202464Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c correlation ID: 80cf0427-c640-486c-a738-9d16fdc0c4e5 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-8fz8f5" with version 4.21.39 on candidate channel Unexpected error: <fmt.wrapError \| 0x17985ea62800>: failed to create HCP cluster cluster-zstream-4-21-8fz8f5: failed waiting for cluster="cluster-zstream-4-21-8fz8f5" in resourcegroup="rg-zstream-upgrade-4-21-wbqzpvwdc6q6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c", "name": "487512e6-43f7-42b5-a694-f4be2585d80c", "status": "Failed", "startTime": "2026-10-28T14:40:20.778612196Z", "endTime": "2026-10-28T15:09:28.776202464Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c correlation ID: 80cf0427-c640-486c-a738-9d16fdc0c4e5 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt Unexpected error: <fmt.wrapError \| 0x27391d09a3a0>: failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-vvl2jlrsdzwm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f", "name": "a5b87783-6a6a-4fee-a31b-7e52a950339f", "status": "Failed", "startTime": "2026-10-28T15:22:08.258068936Z", "endTime": "2026-10-28T15:51:17.638485832Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, community-operators-catalog, control-plane-pki-operator, dns-operator, kube-scheduler, cluster-storage-operator, machine-approver, openshift-apiserver, csi-snapshot-controller-operator, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, kube-controller-manager, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-operators-catalog, openshift-route-controller-manager, ingress-operator, catalog-operator, packageserver, cluster-autoscaler, openshift-controller-manager, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f correlation ID: c155e67f-4293-4e6f-9b93-fb56774a919b client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt Unexpected error: <fmt.wrapError \| 0x27391d09a3a0>: failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-vvl2jlrsdzwm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f", "name": "a5b87783-6a6a-4fee-a31b-7e52a950339f", "status": "Failed", "startTime": "2026-10-28T15:22:08.258068936Z", "endTime": "2026-10-28T15:51:17.638485832Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, community-operators-catalog, control-plane-pki-operator, dns-operator, kube-scheduler, cluster-storage-operator, machine-approver, openshift-apiserver, csi-snapshot-controller-operator, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, kube-controller-manager, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-operators-catalog, openshift-route-controller-manager, ingress-operator, catalog-operator, packageserver, cluster-autoscaler, openshift-controller-manager, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f correlation ID: c155e67f-4293-4e6f-9b93-fb56774a919b client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-dk6562" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xd48edafd640>: failed to create HCP cluster cp-ystream-upgrade-4-21-dk6562: failed waiting for cluster="cp-ystream-upgrade-4-21-dk6562" in resourcegroup="rg-cp-ystream-upgrade-4-21-w6tg4gwbm4zj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70464892-f572-4f2a-b4a6-385fdceed9ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70464892-f572-4f2a-b4a6-385fdceed9ca", "name": "70464892-f572-4f2a-b4a6-385fdceed9ca", "status": "Failed", "startTime": "2026-10-28T14:40:27.209590226Z", "endTime": "2026-10-28T15:09:28.790353292Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70464892-f572-4f2a-b4a6-385fdceed9ca correlation ID: 5f914fbf-789a-44af-8ded-2b56b051ab07 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-dk6562" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xd48edafd640>: failed to create HCP cluster cp-ystream-upgrade-4-21-dk6562: failed waiting for cluster="cp-ystream-upgrade-4-21-dk6562" in resourcegroup="rg-cp-ystream-upgrade-4-21-w6tg4gwbm4zj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70464892-f572-4f2a-b4a6-385fdceed9ca -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70464892-f572-4f2a-b4a6-385fdceed9ca", "name": "70464892-f572-4f2a-b4a6-385fdceed9ca", "status": "Failed", "startTime": "2026-10-28T14:40:27.209590226Z", "endTime": "2026-10-28T15:09:28.790353292Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70464892-f572-4f2a-b4a6-385fdceed9ca correlation ID: 5f914fbf-789a-44af-8ded-2b56b051ab07 client request ID: unavailable occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 13:09:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapErrors \| 0x1f0baa544120>: failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded { msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded", errs: [ <errors.errorString \| 0x1f0baa40b610>{ s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88", }, <fmt.wrapErrors \| 0x1f0baa5440c0>{ msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded", errs: [ <errors.errorString \| 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference), <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapErrors \| 0x1f0baa544120>: failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded { msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded", errs: [ <errors.errorString \| 0x1f0baa40b610>{ s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88", }, <fmt.wrapErrors \| 0x1f0baa5440c0>{ msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded", errs: [ <errors.errorString \| 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference), <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-qxnh9v" with version 4.23.0-ec.0 on candidate channel Unexpected error: <fmt.wrapError \| 0x9c08c475a0>: failed to create HCP cluster cluster-zstream-4-23-qxnh9v: failed waiting for cluster="cluster-zstream-4-23-qxnh9v" in resourcegroup="rg-zstream-upgrade-4-23-lzbzhtjskg9m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286", "name": "49078e61-6846-4cd0-bfa4-7f73457f1286", "status": "Failed", "startTime": "2026-10-28T14:40:13.468188351Z", "endTime": "2026-10-28T15:09:18.606679709Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-image-registry-operator, olm-operator, kube-scheduler, kube-storage-version-migrator, catalog-operator, machine-approver, azure-cloud-controller-manager, cluster-autoscaler, ingress-operator, openshift-controller-manager, packageserver, konnectivity-agent, hosted-cluster-config-operator, redhat-operators-catalog, dns-operator, control-plane-pki-operator, cluster-policy-controller, cluster-network-operator, kube-controller-manager, cluster-node-tuning-operator, openshift-route-controller-manager, olm-collect-profiles, openshift-apiserver, capi-provider, cluster-storage-operator, csi-snapshot-controller-operator, cluster-version-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286 correlation ID: 0f636737-db3c-44fd-a024-8aa2dae006d5 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-qxnh9v" with version 4.23.0-ec.0 on candidate channel Unexpected error: <fmt.wrapError \| 0x9c08c475a0>: failed to create HCP cluster cluster-zstream-4-23-qxnh9v: failed waiting for cluster="cluster-zstream-4-23-qxnh9v" in resourcegroup="rg-zstream-upgrade-4-23-lzbzhtjskg9m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286", "name": "49078e61-6846-4cd0-bfa4-7f73457f1286", "status": "Failed", "startTime": "2026-10-28T14:40:13.468188351Z", "endTime": "2026-10-28T15:09:18.606679709Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-image-registry-operator, olm-operator, kube-scheduler, kube-storage-version-migrator, catalog-operator, machine-approver, azure-cloud-controller-manager, cluster-autoscaler, ingress-operator, openshift-controller-manager, packageserver, konnectivity-agent, hosted-cluster-config-operator, redhat-operators-catalog, dns-operator, control-plane-pki-operator, cluster-policy-controller, cluster-network-operator, kube-controller-manager, cluster-node-tuning-operator, openshift-route-controller-manager, olm-collect-profiles, openshift-apiserver, capi-provider, cluster-storage-operator, csi-snapshot-controller-operator, cluster-version-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286 correlation ID: 0f636737-db3c-44fd-a024-8aa2dae006d5 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0x18fb1fe8a1e0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-rwstnfz2pdpd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8c849820-891e-4b6d-95ef-663bc731334d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8c849820-891e-4b6d-95ef-663bc731334d", "name": "8c849820-891e-4b6d-95ef-663bc731334d", "status": "Failed", "startTime": "2026-10-28T14:39:19.740667918Z", "endTime": "2026-10-28T15:08:28.665210796Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, community-operators-catalog, openshift-route-controller-manager, csi-snapshot-controller-operator, cluster-policy-controller, catalog-operator, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, certified-operators-catalog, machine-approver, hosted-cluster-config-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, kube-controller-manager, cluster-image-registry-operator, olm-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8c849820-891e-4b6d-95ef-663bc731334d correlation ID: 5c6ea318-da7b-4cf0-897b-e3c0738e611d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0x18fb1fe8a1e0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-rwstnfz2pdpd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8c849820-891e-4b6d-95ef-663bc731334d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8c849820-891e-4b6d-95ef-663bc731334d", "name": "8c849820-891e-4b6d-95ef-663bc731334d", "status": "Failed", "startTime": "2026-10-28T14:39:19.740667918Z", "endTime": "2026-10-28T15:08:28.665210796Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, community-operators-catalog, openshift-route-controller-manager, csi-snapshot-controller-operator, cluster-policy-controller, catalog-operator, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, certified-operators-catalog, machine-approver, hosted-cluster-config-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, kube-controller-manager, cluster-image-registry-operator, olm-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8c849820-891e-4b6d-95ef-663bc731334d correlation ID: 5c6ea318-da7b-4cf0-897b-e3c0738e611d client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0x1b9a78cc2c00>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-678vggk8hw4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc", "name": "95bd7f6e-d28c-4575-8824-c4b761c225fc", "status": "Failed", "startTime": "2026-10-28T15:21:43.03617172Z", "endTime": "2026-10-28T15:50:47.66441232Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc correlation ID: 414e22c5-a26a-4c3c-96be-d66c9e15bf90 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0x1b9a78cc2c00>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-678vggk8hw4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc", "name": "95bd7f6e-d28c-4575-8824-c4b761c225fc", "status": "Failed", "startTime": "2026-10-28T15:21:43.03617172Z", "endTime": "2026-10-28T15:50:47.66441232Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc correlation ID: 414e22c5-a26a-4c3c-96be-d66c9e15bf90 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0x166a5ce4a460>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-xfbmh7tnw8dd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da", "name": "e5ff8914-b12e-4cdf-aa40-3986da34c7da", "status": "Failed", "startTime": "2026-10-28T15:21:57.339002209Z", "endTime": "2026-10-28T15:50:57.634345734Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, hosted-cluster-config-operator, azure-cloud-controller-manager, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, dns-operator, certified-operators-catalog, cluster-network-operator, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, community-operators-catalog, machine-approver, kube-controller-manager, kube-scheduler, openshift-controller-manager, ingress-operator, olm-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da correlation ID: 88b2308c-63cd-42b3-b006-499404a750ad client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0x166a5ce4a460>: failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-xfbmh7tnw8dd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da", "name": "e5ff8914-b12e-4cdf-aa40-3986da34c7da", "status": "Failed", "startTime": "2026-10-28T15:21:57.339002209Z", "endTime": "2026-10-28T15:50:57.634345734Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, hosted-cluster-config-operator, azure-cloud-controller-manager, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, dns-operator, certified-operators-catalog, cluster-network-operator, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, community-operators-catalog, machine-approver, kube-controller-manager, kube-scheduler, openshift-controller-manager, ingress-operator, olm-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da correlation ID: 88b2308c-63cd-42b3-b006-499404a750ad client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0x2c4df68131a0>: failed to create HCP cluster ea-listc5c22j: failed waiting for cluster="ea-listc5c22j" in resourcegroup="ea-list-h4pk6vdwpp9q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4641515b-36d7-4de2-8c4a-a2ec882ba078 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4641515b-36d7-4de2-8c4a-a2ec882ba078", "name": "4641515b-36d7-4de2-8c4a-a2ec882ba078", "status": "Failed", "startTime": "2026-10-28T14:41:41.259309811Z", "endTime": "2026-10-28T15:10:48.460536218Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4641515b-36d7-4de2-8c4a-a2ec882ba078 correlation ID: 1b213c52-9e35-46b4-84dc-9a0420bced86 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0x2c4df68131a0>: failed to create HCP cluster ea-listc5c22j: failed waiting for cluster="ea-listc5c22j" in resourcegroup="ea-list-h4pk6vdwpp9q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4641515b-36d7-4de2-8c4a-a2ec882ba078 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4641515b-36d7-4de2-8c4a-a2ec882ba078", "name": "4641515b-36d7-4de2-8c4a-a2ec882ba078", "status": "Failed", "startTime": "2026-10-28T14:41:41.259309811Z", "endTime": "2026-10-28T15:10:48.460536218Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4641515b-36d7-4de2-8c4a-a2ec882ba078 correlation ID: 1b213c52-9e35-46b4-84dc-9a0420bced86 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0x7c22833c460>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2", "name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2", "status": "Failed", "startTime": "2026-10-28T15:21:06.28860999Z", "endTime": "2026-10-28T15:50:07.63308505Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2 correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0x7c22833c460>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2", "name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2", "status": "Failed", "startTime": "2026-10-28T15:21:06.28860999Z", "endTime": "2026-10-28T15:50:07.63308505Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2 correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0x375811d74ca0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-4dpvzp9p5xb8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/520cf7c9-acbc-4761-abda-193db7d8b590 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/520cf7c9-acbc-4761-abda-193db7d8b590", "name": "520cf7c9-acbc-4761-abda-193db7d8b590", "status": "Failed", "startTime": "2026-10-28T13:45:10.07141772Z", "endTime": "2026-10-28T14:14:10.790489596Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/520cf7c9-acbc-4761-abda-193db7d8b590 correlation ID: 1682ab51-6904-4558-9b5e-bab58d966ecb client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0x375811d74ca0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-4dpvzp9p5xb8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/520cf7c9-acbc-4761-abda-193db7d8b590 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/520cf7c9-acbc-4761-abda-193db7d8b590", "name": "520cf7c9-acbc-4761-abda-193db7d8b590", "status": "Failed", "startTime": "2026-10-28T13:45:10.07141772Z", "endTime": "2026-10-28T14:14:10.790489596Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/520cf7c9-acbc-4761-abda-193db7d8b590 correlation ID: 1682ab51-6904-4558-9b5e-bab58d966ecb client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0x3d893d926940>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a", "name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a", "status": "Failed", "startTime": "2026-10-28T14:39:20.447678223Z", "endTime": "2026-10-28T15:08:28.688541915Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0x3d893d926940>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a", "name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a", "status": "Failed", "startTime": "2026-10-28T14:39:20.447678223Z", "endTime": "2026-10-28T15:08:28.688541915Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-tsmb9zwhf6np/cluster-candidate-5-0-qrqt4m should provision Unexpected error: <fmt.wrapError \| 0x848b1c9a440>: failed to create HCP cluster cluster-candidate-5-0-qrqt4m: failed waiting for cluster="cluster-candidate-5-0-qrqt4m" in resourcegroup="rg-candidate-5-0-tsmb9zwhf6np" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd", "name": "ff4d1e16-4c98-4cf9-acfe-6d00538378fd", "status": "Failed", "startTime": "2026-10-28T14:39:51.38832954Z", "endTime": "2026-10-28T15:08:58.636801847Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd correlation ID: 9841ce5e-79da-4501-8301-7fd87b492c29 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-tsmb9zwhf6np/cluster-candidate-5-0-qrqt4m should provision Unexpected error: <fmt.wrapError \| 0x848b1c9a440>: failed to create HCP cluster cluster-candidate-5-0-qrqt4m: failed waiting for cluster="cluster-candidate-5-0-qrqt4m" in resourcegroup="rg-candidate-5-0-tsmb9zwhf6np" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd", "name": "ff4d1e16-4c98-4cf9-acfe-6d00538378fd", "status": "Failed", "startTime": "2026-10-28T14:39:51.38832954Z", "endTime": "2026-10-28T15:08:58.636801847Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd correlation ID: 9841ce5e-79da-4501-8301-7fd87b492c29 client request ID: unavailable occurred Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 13:09:17 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0x24f1c7fcefa0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407", "name": "3ee91092-37d4-4687-8899-086b94094407", "status": "Failed", "startTime": "2026-10-28T13:42:47.010012275Z", "endTime": "2026-10-28T14:11:50.916418144Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407 correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0x24f1c7fcefa0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407", "name": "3ee91092-37d4-4687-8899-086b94094407", "status": "Failed", "startTime": "2026-10-28T13:42:47.010012275Z", "endTime": "2026-10-28T14:11:50.916418144Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407 correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0x22dc0191a120>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-jc4z76tj8b8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a", "name": "91e4eda4-eb0e-4ff2-b22e-65959d67929a", "status": "Failed", "startTime": "2026-10-28T13:43:05.747830881Z", "endTime": "2026-10-28T14:12:10.776073825Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a correlation ID: 93c5577b-f0d7-4fff-80a7-689379e11923 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0x22dc0191a120>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-jc4z76tj8b8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a", "name": "91e4eda4-eb0e-4ff2-b22e-65959d67929a", "status": "Failed", "startTime": "2026-10-28T13:43:05.747830881Z", "endTime": "2026-10-28T14:12:10.776073825Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a correlation ID: 93c5577b-f0d7-4fff-80a7-689379e11923 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x320c1cc145a0>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6", "name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6", "status": "Failed", "startTime": "2026-10-28T13:43:08.173393078Z", "endTime": "2026-10-28T14:12:10.809270873Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6 correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0x320c1cc145a0>: failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6", "name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6", "status": "Failed", "startTime": "2026-10-28T13:43:08.173393078Z", "endTime": "2026-10-28T14:12:10.809270873Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6 correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-1-c9tfj9kmq2zp/cluster-candidate-5-1-kdsdqq should provision Unexpected error: <fmt.wrapError \| 0x158d9065e120>: failed to create HCP cluster cluster-candidate-5-1-kdsdqq: failed waiting for cluster="cluster-candidate-5-1-kdsdqq" in resourcegroup="rg-candidate-5-1-c9tfj9kmq2zp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886", "name": "82df0ed4-6ad4-45af-825d-883fcc05e886", "status": "Failed", "startTime": "2026-10-28T14:40:01.853789521Z", "endTime": "2026-10-28T15:09:08.621567068Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886 correlation ID: bd6e85e8-f244-476d-8e1b-637c7a4beac1 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-1-c9tfj9kmq2zp/cluster-candidate-5-1-kdsdqq should provision Unexpected error: <fmt.wrapError \| 0x158d9065e120>: failed to create HCP cluster cluster-candidate-5-1-kdsdqq: failed waiting for cluster="cluster-candidate-5-1-kdsdqq" in resourcegroup="rg-candidate-5-1-c9tfj9kmq2zp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886", "name": "82df0ed4-6ad4-45af-825d-883fcc05e886", "status": "Failed", "startTime": "2026-10-28T14:40:01.853789521Z", "endTime": "2026-10-28T15:09:08.621567068Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886 correlation ID: bd6e85e8-f244-476d-8e1b-637c7a4beac1 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-8h2wfx with version 4.21.40 Unexpected error: <fmt.wrapError \| 0x10703f7d02c0>: failed to create HCP cluster np-version-upgrade-cluster-8h2wfx: failed waiting for cluster="np-version-upgrade-cluster-8h2wfx" in resourcegroup="rg-np-version-upgrade-8h2wfx-x8cr8nn2t9dm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/46025fd0-85a0-4f95-ac05-96400b1afb05 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/46025fd0-85a0-4f95-ac05-96400b1afb05", "name": "46025fd0-85a0-4f95-ac05-96400b1afb05", "status": "Failed", "startTime": "2026-10-28T15:21:30.261924644Z", "endTime": "2026-10-28T15:50:37.681049101Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/46025fd0-85a0-4f95-ac05-96400b1afb05 correlation ID: e0cd0169-4de0-43d8-a6d4-ceaf4b51be28 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-8h2wfx with version 4.21.40 Unexpected error: <fmt.wrapError \| 0x10703f7d02c0>: failed to create HCP cluster np-version-upgrade-cluster-8h2wfx: failed waiting for cluster="np-version-upgrade-cluster-8h2wfx" in resourcegroup="rg-np-version-upgrade-8h2wfx-x8cr8nn2t9dm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/46025fd0-85a0-4f95-ac05-96400b1afb05 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/46025fd0-85a0-4f95-ac05-96400b1afb05", "name": "46025fd0-85a0-4f95-ac05-96400b1afb05", "status": "Failed", "startTime": "2026-10-28T15:21:30.261924644Z", "endTime": "2026-10-28T15:50:37.681049101Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/46025fd0-85a0-4f95-ac05-96400b1afb05 correlation ID: e0cd0169-4de0-43d8-a6d4-ceaf4b51be28 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0x188b1ffab580>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pptld2m557n2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb", "name": "160dbbc6-24d7-46a4-a575-459e58e52cbb", "status": "Failed", "startTime": "2026-10-28T13:42:52.080654002Z", "endTime": "2026-10-28T14:12:00.912767678Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, azure-cloud-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, community-operators-catalog, machine-approver, openshift-route-controller-manager, kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, ingress-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, control-plane-pki-operator, cluster-version-operator, konnectivity-agent, redhat-operators-catalog, capi-provider, cluster-node-tuning-operator, csi-snapshot-controller-operator, olm-operator, redhat-marketplace-catalog, certified-operators-catalog, catalog-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb correlation ID: b0bdcafa-1d32-4743-b479-c3a60aa49bed client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0x188b1ffab580>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pptld2m557n2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb", "name": "160dbbc6-24d7-46a4-a575-459e58e52cbb", "status": "Failed", "startTime": "2026-10-28T13:42:52.080654002Z", "endTime": "2026-10-28T14:12:00.912767678Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, azure-cloud-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, community-operators-catalog, machine-approver, openshift-route-controller-manager, kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, ingress-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, control-plane-pki-operator, cluster-version-operator, konnectivity-agent, redhat-operators-catalog, capi-provider, cluster-node-tuning-operator, csi-snapshot-controller-operator, olm-operator, redhat-marketplace-catalog, certified-operators-catalog, catalog-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb correlation ID: b0bdcafa-1d32-4743-b479-c3a60aa49bed client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-t7rr5n" with version 5.0.0-rc.6 on candidate channel Unexpected error: <fmt.wrapError \| 0x237ae7f7c240>: failed to create HCP cluster cluster-zstream-5-0-t7rr5n: failed waiting for cluster="cluster-zstream-5-0-t7rr5n" in resourcegroup="rg-zstream-upgrade-5-0-6fp2zwlhw7bz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136", "name": "8907c7e5-9e3b-4204-856f-e49835870136", "status": "Failed", "startTime": "2026-10-28T14:40:08.646687074Z", "endTime": "2026-10-28T15:09:08.445371105Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, packageserver, azure-cloud-controller-manager, dns-operator, openshift-apiserver, capi-provider, kube-controller-manager, cluster-policy-controller, cluster-network-operator, certified-operators-catalog, control-plane-pki-operator, openshift-route-controller-manager, cluster-image-registry-operator, csi-snapshot-controller-operator, catalog-operator, konnectivity-agent, kube-scheduler, cluster-version-operator, ingress-operator, cluster-autoscaler, redhat-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, olm-operator, machine-approver, kube-storage-version-migrator, hosted-cluster-config-operator, community-operators-catalog, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136 correlation ID: 331db847-c2c2-4f64-8fe5-cddb4a33c4cc client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-t7rr5n" with version 5.0.0-rc.6 on candidate channel Unexpected error: <fmt.wrapError \| 0x237ae7f7c240>: failed to create HCP cluster cluster-zstream-5-0-t7rr5n: failed waiting for cluster="cluster-zstream-5-0-t7rr5n" in resourcegroup="rg-zstream-upgrade-5-0-6fp2zwlhw7bz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136", "name": "8907c7e5-9e3b-4204-856f-e49835870136", "status": "Failed", "startTime": "2026-10-28T14:40:08.646687074Z", "endTime": "2026-10-28T15:09:08.445371105Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, packageserver, azure-cloud-controller-manager, dns-operator, openshift-apiserver, capi-provider, kube-controller-manager, cluster-policy-controller, cluster-network-operator, certified-operators-catalog, control-plane-pki-operator, openshift-route-controller-manager, cluster-image-registry-operator, csi-snapshot-controller-operator, catalog-operator, konnectivity-agent, kube-scheduler, cluster-version-operator, ingress-operator, cluster-autoscaler, redhat-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, olm-operator, machine-approver, kube-storage-version-migrator, hosted-cluster-config-operator, community-operators-catalog, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136 correlation ID: 331db847-c2c2-4f64-8fe5-cddb4a33c4cc client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xb05e00c3740>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-xlt7c8hlnpsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b21f4d3-012f-42bb-8427-610849d10abf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b21f4d3-012f-42bb-8427-610849d10abf", "name": "9b21f4d3-012f-42bb-8427-610849d10abf", "status": "Failed", "startTime": "2026-10-28T13:42:51.039046175Z", "endTime": "2026-10-28T14:12:00.781425402Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b21f4d3-012f-42bb-8427-610849d10abf correlation ID: 1ed53069-e0dd-46b7-8d50-006696cb329c client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xb05e00c3740>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-xlt7c8hlnpsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b21f4d3-012f-42bb-8427-610849d10abf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b21f4d3-012f-42bb-8427-610849d10abf", "name": "9b21f4d3-012f-42bb-8427-610849d10abf", "status": "Failed", "startTime": "2026-10-28T13:42:51.039046175Z", "endTime": "2026-10-28T14:12:00.781425402Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b21f4d3-012f-42bb-8427-610849d10abf correlation ID: 1ed53069-e0dd-46b7-8d50-006696cb329c client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-22-28r6sfshkdrv/cluster-candidate-4-22-gcsx6w should provision Unexpected error: <fmt.wrapError \| 0xe993535c600>: failed to create HCP cluster cluster-candidate-4-22-gcsx6w: failed waiting for cluster="cluster-candidate-4-22-gcsx6w" in resourcegroup="rg-candidate-4-22-28r6sfshkdrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2", "name": "626ada50-804b-4cf3-9aac-6865d44b51f2", "status": "Failed", "startTime": "2026-10-28T14:39:47.572236196Z", "endTime": "2026-10-28T15:08:48.605120634Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2 correlation ID: 8c0ce41d-733b-4d14-b437-6022b87a6a82 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-22-28r6sfshkdrv/cluster-candidate-4-22-gcsx6w should provision Unexpected error: <fmt.wrapError \| 0xe993535c600>: failed to create HCP cluster cluster-candidate-4-22-gcsx6w: failed waiting for cluster="cluster-candidate-4-22-gcsx6w" in resourcegroup="rg-candidate-4-22-28r6sfshkdrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2", "name": "626ada50-804b-4cf3-9aac-6865d44b51f2", "status": "Failed", "startTime": "2026-10-28T14:39:47.572236196Z", "endTime": "2026-10-28T15:08:48.605120634Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2 correlation ID: 8c0ce41d-733b-4d14-b437-6022b87a6a82 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x6733e84edc0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dwrfj8hlrbzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/da5198b1-40ba-4a11-8a73-835a5572147c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/da5198b1-40ba-4a11-8a73-835a5572147c", "name": "da5198b1-40ba-4a11-8a73-835a5572147c", "status": "Failed", "startTime": "2026-10-28T14:30:47.364921948Z", "endTime": "2026-10-28T14:59:48.562085294Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, community-operators-catalog, kube-controller-manager, control-plane-pki-operator, dns-operator, olm-operator, hosted-cluster-config-operator, catalog-operator, machine-approver, openshift-controller-manager, ingress-operator, konnectivity-agent, kube-storage-version-migrator, redhat-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, csi-snapshot-controller-operator, azure-cloud-controller-manager, cluster-storage-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/da5198b1-40ba-4a11-8a73-835a5572147c correlation ID: 9056b230-c338-46fd-8ea6-be84f17e39de client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x6733e84edc0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dwrfj8hlrbzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/da5198b1-40ba-4a11-8a73-835a5572147c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/da5198b1-40ba-4a11-8a73-835a5572147c", "name": "da5198b1-40ba-4a11-8a73-835a5572147c", "status": "Failed", "startTime": "2026-10-28T14:30:47.364921948Z", "endTime": "2026-10-28T14:59:48.562085294Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, community-operators-catalog, kube-controller-manager, control-plane-pki-operator, dns-operator, olm-operator, hosted-cluster-config-operator, catalog-operator, machine-approver, openshift-controller-manager, ingress-operator, konnectivity-agent, kube-storage-version-migrator, redhat-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, csi-snapshot-controller-operator, azure-cloud-controller-manager, cluster-storage-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/da5198b1-40ba-4a11-8a73-835a5572147c correlation ID: 9056b230-c338-46fd-8ea6-be84f17e39de client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x359d41718240>: failed to create HCP cluster np-dg-minor-9nszrs: failed waiting for cluster="np-dg-minor-9nszrs" in resourcegroup="rg-np-dg-minor-9nszrs-cwrbzp2nmgh6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/16ac79cb-f3da-4d2b-9e23-05192317b62b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/16ac79cb-f3da-4d2b-9e23-05192317b62b", "name": "16ac79cb-f3da-4d2b-9e23-05192317b62b", "status": "Failed", "startTime": "2026-10-28T15:21:43.861263865Z", "endTime": "2026-10-28T15:50:47.674486588Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, packageserver, machine-approver, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-network-operator, community-operators-catalog, kube-controller-manager, kube-scheduler, cluster-policy-controller, hosted-cluster-config-operator, azure-cloud-controller-manager, kube-storage-version-migrator, ingress-operator, cluster-image-registry-operator, control-plane-pki-operator, catalog-operator, redhat-operators-catalog, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-version-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/16ac79cb-f3da-4d2b-9e23-05192317b62b correlation ID: 34d201c0-84a7-4c87-ac68-3d8fc5f5cfd8 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x359d41718240>: failed to create HCP cluster np-dg-minor-9nszrs: failed waiting for cluster="np-dg-minor-9nszrs" in resourcegroup="rg-np-dg-minor-9nszrs-cwrbzp2nmgh6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/16ac79cb-f3da-4d2b-9e23-05192317b62b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/16ac79cb-f3da-4d2b-9e23-05192317b62b", "name": "16ac79cb-f3da-4d2b-9e23-05192317b62b", "status": "Failed", "startTime": "2026-10-28T15:21:43.861263865Z", "endTime": "2026-10-28T15:50:47.674486588Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, packageserver, machine-approver, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-network-operator, community-operators-catalog, kube-controller-manager, kube-scheduler, cluster-policy-controller, hosted-cluster-config-operator, azure-cloud-controller-manager, kube-storage-version-migrator, ingress-operator, cluster-image-registry-operator, control-plane-pki-operator, catalog-operator, redhat-operators-catalog, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-version-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/16ac79cb-f3da-4d2b-9e23-05192317b62b correlation ID: 34d201c0-84a7-4c87-ac68-3d8fc5f5cfd8 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-p5rwk6" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x18410323d680>: failed to create HCP cluster cp-ystream-upgrade-4-23-p5rwk6: failed waiting for cluster="cp-ystream-upgrade-4-23-p5rwk6" in resourcegroup="rg-cp-ystream-upgrade-4-23-89hkwz6drhgn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930", "name": "c2309fcd-9ee5-4824-b941-841a87860930", "status": "Failed", "startTime": "2026-10-28T14:40:40.163664046Z", "endTime": "2026-10-28T15:09:48.585939528Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930 correlation ID: 7ec55f9b-2f4a-4e1d-8780-4867568c3427 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-p5rwk6" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x18410323d680>: failed to create HCP cluster cp-ystream-upgrade-4-23-p5rwk6: failed waiting for cluster="cp-ystream-upgrade-4-23-p5rwk6" in resourcegroup="rg-cp-ystream-upgrade-4-23-89hkwz6drhgn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930", "name": "c2309fcd-9ee5-4824-b941-841a87860930", "status": "Failed", "startTime": "2026-10-28T14:40:40.163664046Z", "endTime": "2026-10-28T15:09:48.585939528Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930 correlation ID: 7ec55f9b-2f4a-4e1d-8780-4867568c3427 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.22 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x3c07dd926a00>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-c8lbmc69l4g4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0e1488e0-58a2-4e7f-8b4c-81e77c0439c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0e1488e0-58a2-4e7f-8b4c-81e77c0439c5", "name": "0e1488e0-58a2-4e7f-8b4c-81e77c0439c5", "status": "Failed", "startTime": "2026-10-28T13:42:39.516281954Z", "endTime": "2026-10-28T14:11:40.788662196Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, cluster-network-operator, konnectivity-agent, openshift-route-controller-manager, dns-operator, packageserver, catalog-operator, redhat-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, machine-approver, cluster-node-tuning-operator, ingress-operator, openshift-controller-manager, cluster-version-operator, cluster-image-registry-operator, olm-operator, cluster-policy-controller, certified-operators-catalog, kube-storage-version-migrator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0e1488e0-58a2-4e7f-8b4c-81e77c0439c5 correlation ID: a3f2063d-795c-43da-9375-df07755bc07d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x3c07dd926a00>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-c8lbmc69l4g4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0e1488e0-58a2-4e7f-8b4c-81e77c0439c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0e1488e0-58a2-4e7f-8b4c-81e77c0439c5", "name": "0e1488e0-58a2-4e7f-8b4c-81e77c0439c5", "status": "Failed", "startTime": "2026-10-28T13:42:39.516281954Z", "endTime": "2026-10-28T14:11:40.788662196Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, cluster-network-operator, konnectivity-agent, openshift-route-controller-manager, dns-operator, packageserver, catalog-operator, redhat-operators-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, machine-approver, cluster-node-tuning-operator, ingress-operator, openshift-controller-manager, cluster-version-operator, cluster-image-registry-operator, olm-operator, cluster-policy-controller, certified-operators-catalog, kube-storage-version-migrator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0e1488e0-58a2-4e7f-8b4c-81e77c0439c5 correlation ID: a3f2063d-795c-43da-9375-df07755bc07d client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-dxn5f9" at install version 5.0 Unexpected error: <fmt.wrapError \| 0x11aab681540>: failed to create HCP cluster cp-ystream-upgrade-5-1-dxn5f9: failed waiting for cluster="cp-ystream-upgrade-5-1-dxn5f9" in resourcegroup="rg-cp-ystream-upgrade-5-1-v52ld4h7f9q6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1ee401ab-408e-486f-9c14-dc1be569aeb4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1ee401ab-408e-486f-9c14-dc1be569aeb4", "name": "1ee401ab-408e-486f-9c14-dc1be569aeb4", "status": "Failed", "startTime": "2026-10-28T14:40:36.496002206Z", "endTime": "2026-10-28T15:09:38.615736159Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1ee401ab-408e-486f-9c14-dc1be569aeb4 correlation ID: 41d439a4-d8ea-4b66-a4fa-cd570ec527dc client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-dxn5f9" at install version 5.0 Unexpected error: <fmt.wrapError \| 0x11aab681540>: failed to create HCP cluster cp-ystream-upgrade-5-1-dxn5f9: failed waiting for cluster="cp-ystream-upgrade-5-1-dxn5f9" in resourcegroup="rg-cp-ystream-upgrade-5-1-v52ld4h7f9q6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1ee401ab-408e-486f-9c14-dc1be569aeb4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1ee401ab-408e-486f-9c14-dc1be569aeb4", "name": "1ee401ab-408e-486f-9c14-dc1be569aeb4", "status": "Failed", "startTime": "2026-10-28T14:40:36.496002206Z", "endTime": "2026-10-28T15:09:38.615736159Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1ee401ab-408e-486f-9c14-dc1be569aeb4 correlation ID: 41d439a4-d8ea-4b66-a4fa-cd570ec527dc client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0x33d30568cba0>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428", "name": "30b1eeed-8a58-491f-a679-903582bd2428", "status": "Failed", "startTime": "2026-10-28T15:21:06.381277659Z", "endTime": "2026-10-28T15:50:07.62996819Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428 correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0x33d30568cba0>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428", "name": "30b1eeed-8a58-491f-a679-903582bd2428", "status": "Failed", "startTime": "2026-10-28T15:21:06.381277659Z", "endTime": "2026-10-28T15:50:07.62996819Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428 correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0x30c5485c87e0>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3", "name": "4b718d49-0561-4617-ab78-21f445a937c3", "status": "Failed", "startTime": "2026-10-28T13:43:47.79023892Z", "endTime": "2026-10-28T14:12:50.828658903Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3 correlation ID: 14e2f634-75d0-4322-8234-7441ccabc009 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0x30c5485c87e0>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3", "name": "4b718d49-0561-4617-ab78-21f445a937c3", "status": "Failed", "startTime": "2026-10-28T13:43:47.79023892Z", "endTime": "2026-10-28T14:12:50.828658903Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3 correlation ID: 14e2f634-75d0-4322-8234-7441ccabc009 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x3d84c606700>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-gq7pn8p46nb6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15", "name": "8792581c-3981-4bcf-b8a1-ffd33680ce15", "status": "Failed", "startTime": "2026-10-28T13:42:50.808216413Z", "endTime": "2026-10-28T14:11:50.825999726Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 24m17s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15 correlation ID: 331b0b35-9b61-4406-b1d0-4fe22089cb00 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x3d84c606700>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-gq7pn8p46nb6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15", "name": "8792581c-3981-4bcf-b8a1-ffd33680ce15", "status": "Failed", "startTime": "2026-10-28T13:42:50.808216413Z", "endTime": "2026-10-28T14:11:50.825999726Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 24m17s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15 correlation ID: 331b0b35-9b61-4406-b1d0-4fe22089cb00 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40 Unexpected error: <fmt.wrapError \| 0xbe0fc68500>: failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751", "name": "9e35dee8-3994-4ba7-9234-74a03d742751", "status": "Failed", "startTime": "2026-10-28T15:21:15.512939467Z", "endTime": "2026-10-28T15:50:17.639783703Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751 correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40 Unexpected error: <fmt.wrapError \| 0xbe0fc68500>: failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751", "name": "9e35dee8-3994-4ba7-9234-74a03d742751", "status": "Failed", "startTime": "2026-10-28T15:21:15.512939467Z", "endTime": "2026-10-28T15:50:17.639783703Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751 correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0x1bc38a476940>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-78bq6pzzxwrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f", "name": "70aa6c6b-01ba-49fe-9aa7-e8f005a8163f", "status": "Failed", "startTime": "2026-10-28T13:42:57.38659057Z", "endTime": "2026-10-28T14:12:00.88820442Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, csi-snapshot-controller-operator, capi-provider, openshift-apiserver, konnectivity-agent, hosted-cluster-config-operator, cluster-network-operator, ingress-operator, packageserver, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-marketplace-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, catalog-operator, community-operators-catalog, certified-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, kube-controller-manager, kube-scheduler, redhat-operators-catalog, cluster-policy-controller, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f correlation ID: f60a131e-81c7-4e9f-9c27-ffb965259d3b client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0x1bc38a476940>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-78bq6pzzxwrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f", "name": "70aa6c6b-01ba-49fe-9aa7-e8f005a8163f", "status": "Failed", "startTime": "2026-10-28T13:42:57.38659057Z", "endTime": "2026-10-28T14:12:00.88820442Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, csi-snapshot-controller-operator, capi-provider, openshift-apiserver, konnectivity-agent, hosted-cluster-config-operator, cluster-network-operator, ingress-operator, packageserver, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-marketplace-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, catalog-operator, community-operators-catalog, certified-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, kube-controller-manager, kube-scheduler, redhat-operators-catalog, cluster-policy-controller, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f correlation ID: f60a131e-81c7-4e9f-9c27-ffb965259d3b client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xaf67079a580>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-lj5gnxtczbk2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/58216a84-2005-4cee-8574-3ec06d64a912 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/58216a84-2005-4cee-8574-3ec06d64a912", "name": "58216a84-2005-4cee-8574-3ec06d64a912", "status": "Failed", "startTime": "2026-10-28T13:42:46.725416589Z", "endTime": "2026-10-28T14:11:50.908838257Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/58216a84-2005-4cee-8574-3ec06d64a912 correlation ID: da6466cb-9c46-4c0f-a2f1-842af28d09e4 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xaf67079a580>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-lj5gnxtczbk2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/58216a84-2005-4cee-8574-3ec06d64a912 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/58216a84-2005-4cee-8574-3ec06d64a912", "name": "58216a84-2005-4cee-8574-3ec06d64a912", "status": "Failed", "startTime": "2026-10-28T13:42:46.725416589Z", "endTime": "2026-10-28T14:11:50.908838257Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/58216a84-2005-4cee-8574-3ec06d64a912 correlation ID: da6466cb-9c46-4c0f-a2f1-842af28d09e4 client request ID: unavailable occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 13:09:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0x22e10818450>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded", errs: [ <errors.errorString \| 0x22e1036b8e0>{ s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0x22e10818450>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded", errs: [ <errors.errorString \| 0x22e1036b8e0>{ s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x34041508ad40>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-v7v8vtjjfpl9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/607a11e6-eef4-46b1-8d10-f3860ca879cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/607a11e6-eef4-46b1-8d10-f3860ca879cc", "name": "607a11e6-eef4-46b1-8d10-f3860ca879cc", "status": "Failed", "startTime": "2026-10-28T13:42:56.704531592Z", "endTime": "2026-10-28T14:12:00.891835837Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/607a11e6-eef4-46b1-8d10-f3860ca879cc correlation ID: 0319cb8e-5768-4bf6-ad23-11bc5967a1e8 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x34041508ad40>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-v7v8vtjjfpl9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/607a11e6-eef4-46b1-8d10-f3860ca879cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/607a11e6-eef4-46b1-8d10-f3860ca879cc", "name": "607a11e6-eef4-46b1-8d10-f3860ca879cc", "status": "Failed", "startTime": "2026-10-28T13:42:56.704531592Z", "endTime": "2026-10-28T14:12:00.891835837Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/607a11e6-eef4-46b1-8d10-f3860ca879cc correlation ID: 0319cb8e-5768-4bf6-ad23-11bc5967a1e8 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0x38e84538f300>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-gzwwf4ztwf95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/562f9db4-797b-4ac2-824a-b033e3c72f0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/562f9db4-797b-4ac2-824a-b033e3c72f0c", "name": "562f9db4-797b-4ac2-824a-b033e3c72f0c", "status": "Failed", "startTime": "2026-10-28T13:46:50.32793309Z", "endTime": "2026-10-28T14:15:50.815269703Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/562f9db4-797b-4ac2-824a-b033e3c72f0c correlation ID: a06e94a4-6ab6-4185-9bca-5fc095b3763c client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0x38e84538f300>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-gzwwf4ztwf95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/562f9db4-797b-4ac2-824a-b033e3c72f0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/562f9db4-797b-4ac2-824a-b033e3c72f0c", "name": "562f9db4-797b-4ac2-824a-b033e3c72f0c", "status": "Failed", "startTime": "2026-10-28T13:46:50.32793309Z", "endTime": "2026-10-28T14:15:50.815269703Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/562f9db4-797b-4ac2-824a-b033e3c72f0c correlation ID: a06e94a4-6ab6-4185-9bca-5fc095b3763c client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0x32cee6f94b40>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf", "name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf", "status": "Failed", "startTime": "2026-10-28T14:39:19.663998842Z", "endTime": "2026-10-28T15:08:28.451578554Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0x32cee6f94b40>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf", "name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf", "status": "Failed", "startTime": "2026-10-28T14:39:19.663998842Z", "endTime": "2026-10-28T15:08:28.451578554Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xd2fa34b2580>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa", "name": "e56720ec-9cdb-4896-bffe-b107388330fa", "status": "Failed", "startTime": "2026-10-28T15:21:26.535304826Z", "endTime": "2026-10-28T15:50:27.739063015Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xd2fa34b2580>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa", "name": "e56720ec-9cdb-4896-bffe-b107388330fa", "status": "Failed", "startTime": "2026-10-28T15:21:26.535304826Z", "endTime": "2026-10-28T15:50:27.739063015Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44 Unexpected error: <fmt.wrapError \| 0x2f2f6cd22960>: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909", "name": "67af219e-4e40-4a49-ae77-1d482039a909", "status": "Failed", "startTime": "2026-10-28T15:21:37.478622401Z", "endTime": "2026-10-28T15:50:37.723667825Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909 correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44 Unexpected error: <fmt.wrapError \| 0x2f2f6cd22960>: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909", "name": "67af219e-4e40-4a49-ae77-1d482039a909", "status": "Failed", "startTime": "2026-10-28T15:21:37.478622401Z", "endTime": "2026-10-28T15:50:37.723667825Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909 correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0x10694a205e0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-tznn7ncp65rs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0878d65b-0e30-4ed2-bad3-979640744b0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0878d65b-0e30-4ed2-bad3-979640744b0c", "name": "0878d65b-0e30-4ed2-bad3-979640744b0c", "status": "Failed", "startTime": "2026-10-28T15:21:06.463294437Z", "endTime": "2026-10-28T15:50:07.766061264Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, catalog-operator, olm-operator, packageserver, cluster-network-operator, redhat-operators-catalog, community-operators-catalog, cluster-version-operator, openshift-route-controller-manager, dns-operator, certified-operators-catalog, cluster-policy-controller, hosted-cluster-config-operator, cluster-storage-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, ingress-operator, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0878d65b-0e30-4ed2-bad3-979640744b0c correlation ID: fd861975-abd5-431a-85e6-ec6ec36e2032 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0x10694a205e0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-tznn7ncp65rs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0878d65b-0e30-4ed2-bad3-979640744b0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0878d65b-0e30-4ed2-bad3-979640744b0c", "name": "0878d65b-0e30-4ed2-bad3-979640744b0c", "status": "Failed", "startTime": "2026-10-28T15:21:06.463294437Z", "endTime": "2026-10-28T15:50:07.766061264Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, catalog-operator, olm-operator, packageserver, cluster-network-operator, redhat-operators-catalog, community-operators-catalog, cluster-version-operator, openshift-route-controller-manager, dns-operator, certified-operators-catalog, cluster-policy-controller, hosted-cluster-config-operator, cluster-storage-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, ingress-operator, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0878d65b-0e30-4ed2-bad3-979640744b0c correlation ID: fd861975-abd5-431a-85e6-ec6ec36e2032 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001 Unexpected error: <fmt.wrapError \| 0x3b93705ec3c0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-tmcz7jzc8qv4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9ed078ac-98dc-4c85-b217-6c8192a338f1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9ed078ac-98dc-4c85-b217-6c8192a338f1", "name": "9ed078ac-98dc-4c85-b217-6c8192a338f1", "status": "Failed", "startTime": "2026-10-28T14:39:30.728963233Z", "endTime": "2026-10-28T15:08:38.445396808Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9ed078ac-98dc-4c85-b217-6c8192a338f1 correlation ID: 23b7ed7f-ac0f-446d-a8b8-7064395acb17 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001 Unexpected error: <fmt.wrapError \| 0x3b93705ec3c0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-tmcz7jzc8qv4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9ed078ac-98dc-4c85-b217-6c8192a338f1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9ed078ac-98dc-4c85-b217-6c8192a338f1", "name": "9ed078ac-98dc-4c85-b217-6c8192a338f1", "status": "Failed", "startTime": "2026-10-28T14:39:30.728963233Z", "endTime": "2026-10-28T15:08:38.445396808Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9ed078ac-98dc-4c85-b217-6c8192a338f1 correlation ID: 23b7ed7f-ac0f-446d-a8b8-7064395acb17 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x2c5ef3c68380>: failed to create HCP cluster np-downgrade-xtqrzt: failed waiting for cluster="np-downgrade-xtqrzt" in resourcegroup="rg-np-downgrade-xtqrzt-r62kmdslppc2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cc133c6d-1ecb-4233-846e-389835238d32 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cc133c6d-1ecb-4233-846e-389835238d32", "name": "cc133c6d-1ecb-4233-846e-389835238d32", "status": "Failed", "startTime": "2026-10-28T15:21:46.526133596Z", "endTime": "2026-10-28T15:50:47.81681125Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cc133c6d-1ecb-4233-846e-389835238d32 correlation ID: e4a4ba53-062c-4085-8daa-67420223bf34 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x2c5ef3c68380>: failed to create HCP cluster np-downgrade-xtqrzt: failed waiting for cluster="np-downgrade-xtqrzt" in resourcegroup="rg-np-downgrade-xtqrzt-r62kmdslppc2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cc133c6d-1ecb-4233-846e-389835238d32 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cc133c6d-1ecb-4233-846e-389835238d32", "name": "cc133c6d-1ecb-4233-846e-389835238d32", "status": "Failed", "startTime": "2026-10-28T15:21:46.526133596Z", "endTime": "2026-10-28T15:50:47.81681125Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cc133c6d-1ecb-4233-846e-389835238d32 correlation ID: e4a4ba53-062c-4085-8daa-67420223bf34 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0x2a82fa16ec00>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-6pgk2lfq2z5x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/452dde4b-2f11-4bc7-8844-52cf988c2a61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/452dde4b-2f11-4bc7-8844-52cf988c2a61", "name": "452dde4b-2f11-4bc7-8844-52cf988c2a61", "status": "Failed", "startTime": "2026-10-28T13:42:56.742575586Z", "endTime": "2026-10-28T14:12:00.885748443Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/452dde4b-2f11-4bc7-8844-52cf988c2a61 correlation ID: 0ef05e97-5d14-4a27-8e3b-6769d48b8868 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0x2a82fa16ec00>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-6pgk2lfq2z5x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/452dde4b-2f11-4bc7-8844-52cf988c2a61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/452dde4b-2f11-4bc7-8844-52cf988c2a61", "name": "452dde4b-2f11-4bc7-8844-52cf988c2a61", "status": "Failed", "startTime": "2026-10-28T13:42:56.742575586Z", "endTime": "2026-10-28T14:12:00.885748443Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/452dde4b-2f11-4bc7-8844-52cf988c2a61 correlation ID: 0ef05e97-5d14-4a27-8e3b-6769d48b8868 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0x371784ecc8e0>: failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-xk2p7njp5dsf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b", "name": "b13f5e9a-eaea-4399-8cfa-fbd539c1236b", "status": "Failed", "startTime": "2026-10-28T15:21:06.575502401Z", "endTime": "2026-10-28T15:50:07.653220479Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, hosted-cluster-config-operator, certified-operators-catalog, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-marketplace-catalog, machine-approver, kube-controller-manager, dns-operator, cluster-version-operator, openshift-route-controller-manager, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, azure-cloud-controller-manager, olm-operator, community-operators-catalog, ingress-operator, konnectivity-agent, packageserver, cluster-policy-controller, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b correlation ID: 5f2b42c0-ed6b-46a2-99e5-52df9f0918c9 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0x371784ecc8e0>: failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-xk2p7njp5dsf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b", "name": "b13f5e9a-eaea-4399-8cfa-fbd539c1236b", "status": "Failed", "startTime": "2026-10-28T15:21:06.575502401Z", "endTime": "2026-10-28T15:50:07.653220479Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, hosted-cluster-config-operator, certified-operators-catalog, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-marketplace-catalog, machine-approver, kube-controller-manager, dns-operator, cluster-version-operator, openshift-route-controller-manager, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, azure-cloud-controller-manager, olm-operator, community-operators-catalog, ingress-operator, konnectivity-agent, packageserver, cluster-policy-controller, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b correlation ID: 5f2b42c0-ed6b-46a2-99e5-52df9f0918c9 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0x3f31a8d27a80>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-fr6kmzk5bhwz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bf5eb7a7-3ba7-4a13-a695-d60814412b79 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bf5eb7a7-3ba7-4a13-a695-d60814412b79", "name": "bf5eb7a7-3ba7-4a13-a695-d60814412b79", "status": "Failed", "startTime": "2026-10-28T14:39:19.737045251Z", "endTime": "2026-10-28T15:08:28.661748592Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bf5eb7a7-3ba7-4a13-a695-d60814412b79 correlation ID: 27846e27-fa3f-4aac-9b56-7baea8b38233 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0x3f31a8d27a80>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-fr6kmzk5bhwz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bf5eb7a7-3ba7-4a13-a695-d60814412b79 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bf5eb7a7-3ba7-4a13-a695-d60814412b79", "name": "bf5eb7a7-3ba7-4a13-a695-d60814412b79", "status": "Failed", "startTime": "2026-10-28T14:39:19.737045251Z", "endTime": "2026-10-28T15:08:28.661748592Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bf5eb7a7-3ba7-4a13-a695-d60814412b79 correlation ID: 27846e27-fa3f-4aac-9b56-7baea8b38233 client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-bdgcnw" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x10183589cbc0>: failed to create HCP cluster cp-ystream-upgrade-5-0-bdgcnw: failed waiting for cluster="cp-ystream-upgrade-5-0-bdgcnw" in resourcegroup="rg-cp-ystream-upgrade-5-0-cmbc75g7mbtn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71951328-f6e7-4428-bfbf-e48b2270b472 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71951328-f6e7-4428-bfbf-e48b2270b472", "name": "71951328-f6e7-4428-bfbf-e48b2270b472", "status": "Failed", "startTime": "2026-10-28T14:40:58.169629589Z", "endTime": "2026-10-28T15:09:58.460283392Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71951328-f6e7-4428-bfbf-e48b2270b472 correlation ID: 49d0f624-19e9-4c7a-9bf0-8e4f0d5a48ea client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-bdgcnw" at install version 4.22 Unexpected error: <fmt.wrapError \| 0x10183589cbc0>: failed to create HCP cluster cp-ystream-upgrade-5-0-bdgcnw: failed waiting for cluster="cp-ystream-upgrade-5-0-bdgcnw" in resourcegroup="rg-cp-ystream-upgrade-5-0-cmbc75g7mbtn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71951328-f6e7-4428-bfbf-e48b2270b472 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71951328-f6e7-4428-bfbf-e48b2270b472", "name": "71951328-f6e7-4428-bfbf-e48b2270b472", "status": "Failed", "startTime": "2026-10-28T14:40:58.169629589Z", "endTime": "2026-10-28T15:09:58.460283392Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/71951328-f6e7-4428-bfbf-e48b2270b472 correlation ID: 49d0f624-19e9-4c7a-9bf0-8e4f0d5a48ea client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent, kube-controller-manager, kube-scheduler]; provider Microsoft.RedHatOpenShift 13:09:17 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.23 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x186bf96476a0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8", "name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8", "status": "Failed", "startTime": "2026-10-28T13:42:56.996297525Z", "endTime": "2026-10-28T14:12:00.914226401Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8 correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x186bf96476a0>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8", "name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8", "status": "Failed", "startTime": "2026-10-28T13:42:56.996297525Z", "endTime": "2026-10-28T14:12:00.914226401Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8 correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d client request ID: unavailable occurred
02:00:49	1h 58m 21s	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	australiaeast	e2e	failed	n/a	1	e2e (1) ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift 02:00:49 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0x12961691c220>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3", "name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3", "status": "Failed", "startTime": "2026-10-28T02:05:53.148166937Z", "endTime": "2026-10-28T02:34:59.674611282Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3 correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0x12961691c220>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3", "name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3", "status": "Failed", "startTime": "2026-10-28T02:05:53.148166937Z", "endTime": "2026-10-28T02:34:59.674611282Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3 correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1 client request ID: unavailable occurred
01:26:29	2h 23m 0s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	westeurope	n/a	passed	n/a	0	n/a
01:24:17	2h 33m 46s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	brazilsouth	n/a	passed	n/a	0	n/a
01:17:28	2h 16m 46s	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	eastus2	n/a	passed	n/a	0	n/a