fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:172]: failed to resolve test runner public IP
Unexpected error:
    <*fmt.wrapError | 0xc000858480>: 
    failed to query public IP echo service: Get "https://checkip.amazonaws.com": read tcp 172.24.8.136:46186->54.82.178.161:443: read: connection reset by peer
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:172]: failed to resolve test runner public IP
Unexpected error:
    <*fmt.wrapError | 0xc000858480>: 
    failed to query public IP echo service: Get "https://checkip.amazonaws.com": read tcp 172.24.8.136:46186->54.82.178.161:443: read: connection reset by peer
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000f6d380>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-4tsj8j" in resourcegroup="rg-np-noedge-4tsj8j-67tl4d7r7ztd" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "name": "36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "status": "Failed",
      "startTime": "2026-09-27T23:24:00.446121961Z",
      "endTime": "2026-09-27T23:43:03.337836309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000f6d380>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-4tsj8j" in resourcegroup="rg-np-noedge-4tsj8j-67tl4d7r7ztd" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "name": "36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "status": "Failed",
      "startTime": "2026-09-27T23:24:00.446121961Z",
      "endTime": "2026-09-27T23:43:03.337836309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00060efe0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wz5nqx5brl4f" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "name": "3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "status": "Failed",
      "startTime": "2026-09-27T23:20:03.595087025Z",
      "endTime": "2026-09-27T23:39:03.372293439Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-dq426: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-wz5nqx5brl4f--managed/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Network/networkInterfaces/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00060efe0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wz5nqx5brl4f" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "name": "3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "status": "Failed",
      "startTime": "2026-09-27T23:20:03.595087025Z",
      "endTime": "2026-09-27T23:39:03.372293439Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-dq426: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-wz5nqx5brl4f--managed/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Network/networkInterfaces/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T23:23:56Z
  Ended:       2026-09-28T00:09:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j9480704-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j9480704-mgmt-1
Namespace: ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T23:20:54Z
  Ended:       2026-09-27T23:44:54Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j9480704-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j9480704-mgmt-1
Namespace: ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013e0660>: 
    failed to create HCP cluster np-dg-minor-z78ths: failed waiting for cluster="np-dg-minor-z78ths" in resourcegroup="rg-np-dg-minor-z78ths-lxsmmmlnvmjv" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54291c20-e9b4-4070-9657-55e8e053432d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54291c20-e9b4-4070-9657-55e8e053432d",
      "name": "54291c20-e9b4-4070-9657-55e8e053432d",
      "status": "Failed",
      "startTime": "2026-09-27T22:21:31.146099917Z",
      "endTime": "2026-09-27T22:50:31.661815914Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013e0660>: 
    failed to create HCP cluster np-dg-minor-z78ths: failed waiting for cluster="np-dg-minor-z78ths" in resourcegroup="rg-np-dg-minor-z78ths-lxsmmmlnvmjv" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54291c20-e9b4-4070-9657-55e8e053432d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54291c20-e9b4-4070-9657-55e8e053432d",
      "name": "54291c20-e9b4-4070-9657-55e8e053432d",
      "status": "Failed",
      "startTime": "2026-09-27T22:21:31.146099917Z",
      "endTime": "2026-09-27T22:50:31.661815914Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-97fd8t2fpt8h/cluster-candidate-4-23-jvmr6j should provision
Unexpected error:
    <*fmt.wrapError | 0xc00106ed20>: 
    failed to create HCP cluster cluster-candidate-4-23-jvmr6j: failed waiting for cluster="cluster-candidate-4-23-jvmr6j" in resourcegroup="rg-candidate-4-23-97fd8t2fpt8h" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1d51430-01dc-4d6c-9f0a-0b21f000ca8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1d51430-01dc-4d6c-9f0a-0b21f000ca8d",
      "name": "d1d51430-01dc-4d6c-9f0a-0b21f000ca8d",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:36.712987842Z",
      "endTime": "2026-09-27T22:26:36.653416533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-97fd8t2fpt8h/cluster-candidate-4-23-jvmr6j should provision
Unexpected error:
    <*fmt.wrapError | 0xc00106ed20>: 
    failed to create HCP cluster cluster-candidate-4-23-jvmr6j: failed waiting for cluster="cluster-candidate-4-23-jvmr6j" in resourcegroup="rg-candidate-4-23-97fd8t2fpt8h" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1d51430-01dc-4d6c-9f0a-0b21f000ca8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1d51430-01dc-4d6c-9f0a-0b21f000ca8d",
      "name": "d1d51430-01dc-4d6c-9f0a-0b21f000ca8d",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:36.712987842Z",
      "endTime": "2026-09-27T22:26:36.653416533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f1e260>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hhxkjbgbq9fb" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "name": "3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.777648405Z",
      "endTime": "2026-09-27T22:26:17.06664737Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f1e260>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hhxkjbgbq9fb" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "name": "3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.777648405Z",
      "endTime": "2026-09-27T22:26:17.06664737Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de2a20>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-tcmg5wddbpw5" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "name": "75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:31.92972128Z",
      "endTime": "2026-09-27T22:26:36.766928511Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de2a20>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-tcmg5wddbpw5" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "name": "75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:31.92972128Z",
      "endTime": "2026-09-27T22:26:36.766928511Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000bc7480>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fzbdj6z9fvsp" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ae4b814-345a-401a-ad49-b8ffd613eb42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ae4b814-345a-401a-ad49-b8ffd613eb42",
      "name": "7ae4b814-345a-401a-ad49-b8ffd613eb42",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:27.476971207Z",
      "endTime": "2026-09-27T22:26:36.91910306Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000bc7480>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fzbdj6z9fvsp" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ae4b814-345a-401a-ad49-b8ffd613eb42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ae4b814-345a-401a-ad49-b8ffd613eb42",
      "name": "7ae4b814-345a-401a-ad49-b8ffd613eb42",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:27.476971207Z",
      "endTime": "2026-09-27T22:26:36.91910306Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a74820>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-d7zqqvsfl6cw" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f90451ae-33e2-4eab-9a3e-26e39b48aef2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f90451ae-33e2-4eab-9a3e-26e39b48aef2",
      "name": "f90451ae-33e2-4eab-9a3e-26e39b48aef2",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:31.347881728Z",
      "endTime": "2026-09-27T22:26:36.956451886Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a74820>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-d7zqqvsfl6cw" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f90451ae-33e2-4eab-9a3e-26e39b48aef2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f90451ae-33e2-4eab-9a3e-26e39b48aef2",
      "name": "f90451ae-33e2-4eab-9a3e-26e39b48aef2",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:31.347881728Z",
      "endTime": "2026-09-27T22:26:36.956451886Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000f767a0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-tlzxr8jmz7kk" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "name": "4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.95420119Z",
      "endTime": "2026-09-27T22:26:17.083195861Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000f767a0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-tlzxr8jmz7kk" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "name": "4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.95420119Z",
      "endTime": "2026-09-27T22:26:17.083195861Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000c9f6a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-tfnpjqsqltmz" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef11fb4d-c122-4f3d-8662-7a6be0a63439
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef11fb4d-c122-4f3d-8662-7a6be0a63439",
      "name": "ef11fb4d-c122-4f3d-8662-7a6be0a63439",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:17.555091008Z",
      "endTime": "2026-09-27T22:26:17.063491391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000c9f6a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-tfnpjqsqltmz" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef11fb4d-c122-4f3d-8662-7a6be0a63439
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef11fb4d-c122-4f3d-8662-7a6be0a63439",
      "name": "ef11fb4d-c122-4f3d-8662-7a6be0a63439",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:17.555091008Z",
      "endTime": "2026-09-27T22:26:17.063491391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 13 time(s) (6 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:06:28Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T22:06:29Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-27T22:07:30Z
  Ended:       2026-09-27T22:48:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-27T22:07:30Z
  Ended:       2026-09-27T22:48:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-27T22:09:27Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-27T22:10:30Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd-0 has been in a non-ready state for longer than 5 minutes.

alert fired 12 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-27T22:19:32Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-27T22:19:32Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-27T22:20:28Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-27T22:20:28Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:08:16Z
  Ended:       2026-09-27T23:04:17Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j2557696-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2557696-mgmt-2
Namespace: ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:05:16Z
  Ended:       2026-09-27T22:39:15Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j2557696-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2557696-mgmt-2
Namespace: ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000fa8060>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-kcdxbmxql2zw" to finish creating: GET https://rp.j8939648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1",
      "name": "4c589deb-304d-4351-af73-1056e66b3ff1",
      "status": "Failed",
      "startTime": "2026-09-27T20:35:22.462990652Z",
      "endTime": "2026-09-27T20:54:31.807536747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000fa8060>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-kcdxbmxql2zw" to finish creating: GET https://rp.j8939648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1",
      "name": "4c589deb-304d-4351-af73-1056e66b3ff1",
      "status": "Failed",
      "startTime": "2026-09-27T20:35:22.462990652Z",
      "endTime": "2026-09-27T20:54:31.807536747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T20:30:25Z
  Ended:       2026-09-27T21:00:21Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster

Firing 2:
  State:       Resolved
  Started:     2026-09-27T20:30:25Z
  Ended:       2026-09-27T20:55:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T20:33:23Z
  Ended:       2026-09-27T21:25:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster

Firing 2:
  State:       Resolved
  Started:     2026-09-27T20:33:23Z
  Ended:       2026-09-27T21:20:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000f972a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-t7rmzspt8s2k" to finish creating: GET https://rp.j6124032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "name": "0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "status": "Failed",
      "startTime": "2026-09-27T20:26:25.730010902Z",
      "endTime": "2026-09-27T20:45:26.626992422Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k: Failed: interfaces failed to create or update. err: failed to get existing resource nodepool-delete-t7rmzspt8s2k--managed/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-t7rmzspt8s2k--managed/providers/Microsoft.Network/networkInterfaces/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' seconds after additiona... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000f972a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-t7rmzspt8s2k" to finish creating: GET https://rp.j6124032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "name": "0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "status": "Failed",
      "startTime": "2026-09-27T20:26:25.730010902Z",
      "endTime": "2026-09-27T20:45:26.626992422Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k: Failed: interfaces failed to create or update. err: failed to get existing resource nodepool-delete-t7rmzspt8s2k--managed/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-t7rmzspt8s2k--managed/providers/Microsoft.Network/networkInterfaces/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' seconds after additiona... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:30:48Z
  Ended:       2026-09-27T20:20:45Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j1590528-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3hrgvt17an29u42rn5461sqrrkuc36-agg-cluster", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j1590528-mgmt-2
Namespace: ocm-arohcpci01-2t3hrgvt17an29u42rn5461sqrrkuc36-agg-cluster

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:28:50Z
  Ended:       2026-09-27T20:12:49Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j1590528-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j1590528-mgmt-1
Namespace: ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:27:47Z
  Ended:       2026-09-27T19:55:47Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j1590528-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3hrgvt17an29u42rn5461sqrrkuc36-agg-cluster", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j1590528-mgmt-2
Namespace: ocm-arohcpci01-2t3hrgvt17an29u42rn5461sqrrkuc36-agg-cluster

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:25:52Z
  Ended:       2026-09-27T19:47:50Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j1590528-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j1590528-mgmt-1
Namespace: ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:29:26Z
  Ended:       2026-09-27T20:13:26Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j2756608-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3hq6389up01vpn0d0ugl7ajsd0kjv0-q8g1r5a9g2q9m7l", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2756608-mgmt-1
Namespace: ocm-arohcpci01-2t3hq6389up01vpn0d0ugl7ajsd0kjv0-q8g1r5a9g2q9m7l

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:26:26Z
  Ended:       2026-09-27T19:48:27Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j2756608-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3hq6389up01vpn0d0ugl7ajsd0kjv0-q8g1r5a9g2q9m7l", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2756608-mgmt-1
Namespace: ocm-arohcpci01-2t3hq6389up01vpn0d0ugl7ajsd0kjv0-q8g1r5a9g2q9m7l

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T18:49:46Z
  Ended:       2026-09-27T18:56:46Z
  Severity:    Sev3
  Labels:      alertname="FleetControllerRetryHotLoop", cluster="ci01-j8175232-svc", component="fleet", name="nodepoolcontroller", region="westus3", severity="warning"
  Description: Fleet controller workqueue nodepoolcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:133]: failed to get admin REST config for cluster "cluster-candidate-4-21-cscwdf"
Unexpected error:
    <*fmt.wrapError | 0xc0001673c0>: 
    failed waiting for hcpCluster="cluster-candidate-4-21-cscwdf" in resourcegroup="rg-candidate-4-21-xbwd2trtds6r" to finish getting creds: GET https://rp.j6653824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:133]: failed to get admin REST config for cluster "cluster-candidate-4-21-cscwdf"
Unexpected error:
    <*fmt.wrapError | 0xc0001673c0>: 
    failed waiting for hcpCluster="cluster-candidate-4-21-cscwdf" in resourcegroup="rg-candidate-4-21-xbwd2trtds6r" to finish getting creds: GET https://rp.j6653824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T18:07:35Z
  Ended:       2026-09-27T18:36:37Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j6653824-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3gm692dov66okrgiqv3gkivt7lh79a-agg-cluster", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j6653824-mgmt-1
Namespace: ocm-arohcpci01-2t3gm692dov66okrgiqv3gkivt7lh79a-agg-cluster

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T18:10:36Z
  Ended:       2026-09-27T19:01:36Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j6653824-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3gm692dov66okrgiqv3gkivt7lh79a-agg-cluster", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j6653824-mgmt-1
Namespace: ocm-arohcpci01-2t3gm692dov66okrgiqv3gkivt7lh79a-agg-cluster

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T17:37:18Z
  Ended:       2026-09-27T17:43:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1132032-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3g642vb3ovmhl4qiumvarv9qifaek4-c6e1u6h4n5k3a7j", pod="router-5584fc994b-5x9hp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3g642vb3ovmhl4qiumvarv9qifaek4-c6e1u6h4n5k3a7j/router-5584fc994b-5x9hp has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T18:00:15Z
  Ended:       2026-09-27T18:06:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1132032-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1132032-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00089fe00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-25nvltvpwf84" to finish creating: GET https://rp.j5652992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "name": "6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "status": "Failed",
      "startTime": "2026-09-27T17:01:59.070822701Z",
      "endTime": "2026-09-27T17:21:03.436757642Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-25l2v-tmrtf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-25l2v-tmrtf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00089fe00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-25nvltvpwf84" to finish creating: GET https://rp.j5652992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "name": "6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "status": "Failed",
      "startTime": "2026-09-27T17:01:59.070822701Z",
      "endTime": "2026-09-27T17:21:03.436757642Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-25l2v-tmrtf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-25l2v-tmrtf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T16:57:42Z
  Ended:       2026-09-27T17:30:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5652992-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5652992-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T17:14:22Z
  Ended:       2026-09-27T17:26:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j5652992-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j5652992-mgmt-2
Namespace: ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T17:21:23Z
  Ended:       2026-09-27T17:51:27Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j5652992-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j5652992-mgmt-2
Namespace: ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T14:40:20Z
  Ended:       2026-09-27T14:57:23Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5467008-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T14:38:39Z
  Ended:       2026-09-27T15:26:38Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j5467008-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3di9vg5945pk406769u1t7vrpjin1b-v20261001", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j5467008-mgmt-2
Namespace: ocm-arohcpci01-2t3di9vg5945pk406769u1t7vrpjin1b-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T14:35:35Z
  Ended:       2026-09-27T15:01:36Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j5467008-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3di9vg5945pk406769u1t7vrpjin1b-v20261001", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j5467008-mgmt-2
Namespace: ocm-arohcpci01-2t3di9vg5945pk406769u1t7vrpjin1b-v20261001

alert fired 8 time(s) (1 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T14:26:50Z
  Ended:       2026-09-27T14:32:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5453056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3d20g44qhuupsokqk6foneks2ohpk7-agg-cluster", pod="kube-apiserver-857c58cf4-hl9vw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3d20g44qhuupsokqk6foneks2ohpk7-agg-cluster/kube-apiserver-857c58cf4-hl9vw has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000676bd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000676bd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0029105a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-24vw7c2tw2cd" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-24vw7c2tw2cd, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0029105a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-24vw7c2tw2cd" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-24vw7c2tw2cd, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc001692000>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-cgcqm9ph7z7k" to finish creating: GET https://rp.j9166976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/960cce01-25b6-4e04-aeb2-169c0394ac37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/960cce01-25b6-4e04-aeb2-169c0394ac37",
      "name": "960cce01-25b6-4e04-aeb2-169c0394ac37",
      "status": "Failed",
      "startTime": "2026-09-27T08:48:39.120988577Z",
      "endTime": "2026-09-27T09:07:39.122832689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-m7pgk-gh8dm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-cgcqm9ph7z7k--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-m7pgk-gh8dm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-m7pgk-gh8dm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-cgcqm9ph7z7k--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-m7pgk-gh8dm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc001692000>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-cgcqm9ph7z7k" to finish creating: GET https://rp.j9166976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/960cce01-25b6-4e04-aeb2-169c0394ac37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/960cce01-25b6-4e04-aeb2-169c0394ac37",
      "name": "960cce01-25b6-4e04-aeb2-169c0394ac37",
      "status": "Failed",
      "startTime": "2026-09-27T08:48:39.120988577Z",
      "endTime": "2026-09-27T09:07:39.122832689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-m7pgk-gh8dm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-cgcqm9ph7z7k--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-m7pgk-gh8dm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-m7pgk-gh8dm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-cgcqm9ph7z7k--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-m7pgk-gh8dm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cb7860>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jmzm77" in resourcegroup="rg-np-version-upgrade-jmzm77-gnwj7tdglbmr" to finish creating: GET https://rp.j9166976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2bbfea9-d25c-4222-afe4-d2febbb37c31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2bbfea9-d25c-4222-afe4-d2febbb37c31",
      "name": "f2bbfea9-d25c-4222-afe4-d2febbb37c31",
      "status": "Failed",
      "startTime": "2026-09-27T08:50:17.403277735Z",
      "endTime": "2026-09-27T09:09:19.085896095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-jmzm77-gnwj7tdglbmr-np-upgrade-jm-12b3e238/providers/Microsoft.Compute/virtualMachines/v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-jmzm77-gnwj7tdglbmr-np-upgrade-jm-12b3e238/providers/Microsoft.Compute/virtualMachines/v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cb7860>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jmzm77" in resourcegroup="rg-np-version-upgrade-jmzm77-gnwj7tdglbmr" to finish creating: GET https://rp.j9166976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2bbfea9-d25c-4222-afe4-d2febbb37c31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2bbfea9-d25c-4222-afe4-d2febbb37c31",
      "name": "f2bbfea9-d25c-4222-afe4-d2febbb37c31",
      "status": "Failed",
      "startTime": "2026-09-27T08:50:17.403277735Z",
      "endTime": "2026-09-27T09:09:19.085896095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-jmzm77-gnwj7tdglbmr-np-upgrade-jm-12b3e238/providers/Microsoft.Compute/virtualMachines/v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-jmzm77-gnwj7tdglbmr-np-upgrade-jm-12b3e238/providers/Microsoft.Compute/virtualMachines/v9s3g2z1w8y5m9a-npupgrade-4-20-vbtk2-dsvr7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001793e90>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-99j2d9sd47k2" to finish creating: GET https://rp.j9166976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f398cd92-c2d5-469e-8eca-cf2a30ef86ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f398cd92-c2d5-469e-8eca-cf2a30ef86ac",
      "name": "f398cd92-c2d5-469e-8eca-cf2a30ef86ac",
      "status": "Failed",
      "startTime": "2026-09-27T08:52:25.123568687Z",
      "endTime": "2026-09-27T09:11:28.968750353Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-vvmdb-sgtsq: Failed: availabilitysets failed to create or update. err: failed to get existing resource feature-aggregation-99j2d9sd47k2--managed/2t38glvvr2plvdov72ijndtosji9gdee_agg-cluster-agg-np-a-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-99j2d9sd47k2--managed/providers/Microsoft.Compute/availabilitySets/2t38glvvr2plvdov72ijndtosji9gdee_agg-cluster-agg-np-a-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'ccbf438f-08ec-4e2b-ae4e-219ce7dc3762' exceeded. Please try again after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001793e90>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-99j2d9sd47k2" to finish creating: GET https://rp.j9166976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f398cd92-c2d5-469e-8eca-cf2a30ef86ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f398cd92-c2d5-469e-8eca-cf2a30ef86ac",
      "name": "f398cd92-c2d5-469e-8eca-cf2a30ef86ac",
      "status": "Failed",
      "startTime": "2026-09-27T08:52:25.123568687Z",
      "endTime": "2026-09-27T09:11:28.968750353Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-vvmdb-sgtsq: Failed: availabilitysets failed to create or update. err: failed to get existing resource feature-aggregation-99j2d9sd47k2--managed/2t38glvvr2plvdov72ijndtosji9gdee_agg-cluster-agg-np-a-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-99j2d9sd47k2--managed/providers/Microsoft.Compute/availabilitySets/2t38glvvr2plvdov72ijndtosji9gdee_agg-cluster-agg-np-a-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'ccbf438f-08ec-4e2b-ae4e-219ce7dc3762' exceeded. Please try again after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T08:05:14Z
  Ended:       2026-09-27T08:31:16Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j0172672-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0172672-mgmt-1
Namespace: ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T08:29:34Z
  Ended:       2026-09-27T08:35:38Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0172672-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0172672-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T08:08:19Z
  Ended:       2026-09-27T08:56:14Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j0172672-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0172672-mgmt-1
Namespace: ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002177290>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002177290>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T06:19:21Z
  Ended:       2026-09-27T06:25:22Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9712256-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9712256-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000906ee0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5n7q2p9clx8c" to finish creating: GET https://rp.j2686976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fec5206a-4b8c-473b-a0b3-7fd919567d02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fec5206a-4b8c-473b-a0b3-7fd919567d02",
      "name": "fec5206a-4b8c-473b-a0b3-7fd919567d02",
      "status": "Failed",
      "startTime": "2026-09-27T05:55:45.853039376Z",
      "endTime": "2026-09-27T06:14:46.209421265Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o4g6s6i3z3o6t4d-np-1-gj46m-mnj9b: Failed: availabilitysets failed to create or update. err: failed to get existing resource autoscaling-cluster-5n7q2p9clx8c--managed/2t35su4i1486ad8biqf1t6jj8eevkeio_o4g6s6i3z3o6t4d-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5n7q2p9clx8c--managed/providers/Microsoft.Compute/availabilitySets/2t35su4i1486ad8biqf1t6jj8eevkeio_o4g6s6i3z3o6t4d-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000906ee0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5n7q2p9clx8c" to finish creating: GET https://rp.j2686976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fec5206a-4b8c-473b-a0b3-7fd919567d02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fec5206a-4b8c-473b-a0b3-7fd919567d02",
      "name": "fec5206a-4b8c-473b-a0b3-7fd919567d02",
      "status": "Failed",
      "startTime": "2026-09-27T05:55:45.853039376Z",
      "endTime": "2026-09-27T06:14:46.209421265Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o4g6s6i3z3o6t4d-np-1-gj46m-mnj9b: Failed: availabilitysets failed to create or update. err: failed to get existing resource autoscaling-cluster-5n7q2p9clx8c--managed/2t35su4i1486ad8biqf1t6jj8eevkeio_o4g6s6i3z3o6t4d-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5n7q2p9clx8c--managed/providers/Microsoft.Compute/availabilitySets/2t35su4i1486ad8biqf1t6jj8eevkeio_o4g6s6i3z3o6t4d-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T05:56:50Z
  Ended:       2026-09-27T06:14:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j2686976-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2686976-mgmt-2
Namespace: ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T06:17:28Z
  Ended:       2026-09-27T06:26:29Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2686976-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2686976-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T05:59:52Z
  Ended:       2026-09-27T06:43:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j2686976-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2686976-mgmt-2
Namespace: ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T05:55:51Z
  Ended:       2026-09-27T06:20:52Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2686976-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2686976-mgmt-2
Namespace: ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T05:58:53Z
  Ended:       2026-09-27T06:45:53Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j2686976-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2686976-mgmt-2
Namespace: ocm-arohcpci01-2t35ss5fcakhougflv038gatblofrv6a-v20261001

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc0007a2980>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-5fskdvpkxfwb" to finish creating: GET https://rp.j7860224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "name": "2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "status": "Failed",
      "startTime": "2026-09-27T05:48:02.556455592Z",
      "endTime": "2026-09-27T06:07:09.276038359Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2: Failed: interfaces failed to create or update. err: failed to get existing resource admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/providers/Microsoft.Network/networkInterfaces/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceeded. Please try again after '10' seconds after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc0007a2980>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-5fskdvpkxfwb" to finish creating: GET https://rp.j7860224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "name": "2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "status": "Failed",
      "startTime": "2026-09-27T05:48:02.556455592Z",
      "endTime": "2026-09-27T06:07:09.276038359Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2: Failed: interfaces failed to create or update. err: failed to get existing resource admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/providers/Microsoft.Network/networkInterfaces/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceeded. Please try again after '10' seconds after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T05:53:03Z
  Ended:       2026-09-27T06:39:04Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j7860224-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t35rsiharub78pcihjajhlq044jn0tm-p6u8t9u3z0f3e0k", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j7860224-mgmt-1
Namespace: ocm-arohcpci01-2t35rsiharub78pcihjajhlq044jn0tm-p6u8t9u3z0f3e0k

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T05:50:05Z
  Ended:       2026-09-27T06:14:01Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j7860224-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t35rsiharub78pcihjajhlq044jn0tm-p6u8t9u3z0f3e0k", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j7860224-mgmt-1
Namespace: ocm-arohcpci01-2t35rsiharub78pcihjajhlq044jn0tm-p6u8t9u3z0f3e0k

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T06:16:41Z
  Ended:       2026-09-27T06:22:42Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3477888-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3477888-svc).

time=2026-09-27T04:55:22.142Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2 description="Step kusto-grant-ingest\n  Kind: ARM\n  Template: templates/kusto-grant-ingest.bicep\n  Parameters: configurations/kusto-mgmt-ingest.tmpl.bicepparam"
time=2026-09-27T04:55:22.662Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2
time=2026-09-27T04:55:23.741Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2 deployment=2e905e8738711f49918794c12ee7bdb55d7d9202820d0f6abb23829107a4ce80 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8656896-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2e905e8738711f49918794c12ee7bdb55d7d9202820d0f6abb23829107a4ce80
time=2026-09-27T04:59:54.839Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8656896-mgmt-2/providers/Microsoft.Resources/deployments/2e905e8738711f49918794c12ee7bdb55d7d9202820d0f6abb23829107a4ce80/operationStatuses/08584111219623969246\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/grantKusto-hcp-ja2yqt7pclopi\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-kusto-us/providers/Microsoft.Kusto/clusters/hcp-dev-us-2/databases/HostedControlPlaneLogs/principalAssignments/grant-b6fd691c-fba0-50f9-bb53-13af017771ec\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Failed\\\",\\r\\n                \\\"message\\\": \\\"Internal Server Error\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T01:21:42Z
  Ended:       2026-09-27T01:35:45Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8776064-svc", component="backend", name="datadump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T01:22:43Z
  Ended:       2026-09-27T01:37:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8776064-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00116fc20>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kj25nlz5xd2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "name": "f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "status": "Failed",
      "startTime": "2026-09-27T07:29:57.086711017Z",
      "endTime": "2026-09-27T07:48:57.003548378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00116fc20>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kj25nlz5xd2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "name": "f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "status": "Failed",
      "startTime": "2026-09-27T07:29:57.086711017Z",
      "endTime": "2026-09-27T07:48:57.003548378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000bfe8e0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-pz8l8grcdj98" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55967661-6f28-4965-8bcc-cb1836294e43
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55967661-6f28-4965-8bcc-cb1836294e43",
      "name": "55967661-6f28-4965-8bcc-cb1836294e43",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:37.37348914Z",
      "endTime": "2026-09-27T23:31:45.970076258Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000bfe8e0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-pz8l8grcdj98" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55967661-6f28-4965-8bcc-cb1836294e43
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55967661-6f28-4965-8bcc-cb1836294e43",
      "name": "55967661-6f28-4965-8bcc-cb1836294e43",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:37.37348914Z",
      "endTime": "2026-09-27T23:31:45.970076258Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc002ba0720>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4pp72c" resourcegroup="rg-np-dg-minor-4pp72c-t57mnh5thxv5": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4pp72c-t57mnh5thxv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4pp72c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc002ba0720>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4pp72c" resourcegroup="rg-np-dg-minor-4pp72c-t57mnh5thxv5": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4pp72c-t57mnh5thxv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4pp72c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0012944a0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jgmdt9882d2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/468a566e-d147-4e1c-96c1-38bddafc0b6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/468a566e-d147-4e1c-96c1-38bddafc0b6f",
      "name": "468a566e-d147-4e1c-96c1-38bddafc0b6f",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:17.0613341Z",
      "endTime": "2026-09-27T23:31:25.968220027Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0012944a0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jgmdt9882d2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/468a566e-d147-4e1c-96c1-38bddafc0b6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/468a566e-d147-4e1c-96c1-38bddafc0b6f",
      "name": "468a566e-d147-4e1c-96c1-38bddafc0b6f",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:17.0613341Z",
      "endTime": "2026-09-27T23:31:25.968220027Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000a81100>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-9zjchk" in resourcegroup="rg-np-version-upgrade-9zjchk-r4lv9m9d98td": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9zjchk-r4lv9m9d98td/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-9zjchk/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000a81100>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-9zjchk" in resourcegroup="rg-np-version-upgrade-9zjchk-r4lv9m9d98td": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9zjchk-r4lv9m9d98td/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-9zjchk/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014b70f8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014b70f8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc0015174b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc0015174b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000b50d60>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tgc7nslklnzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1235efb8-3cf2-4da7-8e61-25e5c2bc0a76
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1235efb8-3cf2-4da7-8e61-25e5c2bc0a76",
      "name": "1235efb8-3cf2-4da7-8e61-25e5c2bc0a76",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:36.662922226Z",
      "endTime": "2026-09-27T23:31:45.970102186Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000b50d60>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tgc7nslklnzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1235efb8-3cf2-4da7-8e61-25e5c2bc0a76
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1235efb8-3cf2-4da7-8e61-25e5c2bc0a76",
      "name": "1235efb8-3cf2-4da7-8e61-25e5c2bc0a76",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:36.662922226Z",
      "endTime": "2026-09-27T23:31:45.970102186Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000eac560>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-b68r69hdkgbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c06c0022-17cf-4599-9905-c52d0766bab4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c06c0022-17cf-4599-9905-c52d0766bab4",
      "name": "c06c0022-17cf-4599-9905-c52d0766bab4",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:11.057895557Z",
      "endTime": "2026-09-27T23:34:15.977577559Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000eac560>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-b68r69hdkgbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c06c0022-17cf-4599-9905-c52d0766bab4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c06c0022-17cf-4599-9905-c52d0766bab4",
      "name": "c06c0022-17cf-4599-9905-c52d0766bab4",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:11.057895557Z",
      "endTime": "2026-09-27T23:34:15.977577559Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-tssqn7" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0008d6f80>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-tssqn7: failed waiting for cluster="cp-ystream-upgrade-5-0-tssqn7" in resourcegroup="rg-cp-ystream-upgrade-5-0-s5ql4kptw4ct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "name": "8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:40.978774448Z",
      "endTime": "2026-09-28T00:26:43.612304644Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-tssqn7" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0008d6f80>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-tssqn7: failed waiting for cluster="cp-ystream-upgrade-5-0-tssqn7" in resourcegroup="rg-cp-ystream-upgrade-5-0-s5ql4kptw4ct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "name": "8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:40.978774448Z",
      "endTime": "2026-09-28T00:26:43.612304644Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f1c7e0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-7gqfj78cg774" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1741b83-b940-4f9f-958f-83b4e796a4f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1741b83-b940-4f9f-958f-83b4e796a4f6",
      "name": "d1741b83-b940-4f9f-958f-83b4e796a4f6",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:18.821083127Z",
      "endTime": "2026-09-28T00:28:23.573800876Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f1c7e0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-7gqfj78cg774" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1741b83-b940-4f9f-958f-83b4e796a4f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1741b83-b940-4f9f-958f-83b4e796a4f6",
      "name": "d1741b83-b940-4f9f-958f-83b4e796a4f6",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:18.821083127Z",
      "endTime": "2026-09-28T00:28:23.573800876Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010585e0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-7h98mb66pjnm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5af845a7-7502-4bd6-897b-19467c8a0ada
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5af845a7-7502-4bd6-897b-19467c8a0ada",
      "name": "5af845a7-7502-4bd6-897b-19467c8a0ada",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:52.078427171Z",
      "endTime": "2026-09-27T23:33:55.976360396Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010585e0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-7h98mb66pjnm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5af845a7-7502-4bd6-897b-19467c8a0ada
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5af845a7-7502-4bd6-897b-19467c8a0ada",
      "name": "5af845a7-7502-4bd6-897b-19467c8a0ada",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:52.078427171Z",
      "endTime": "2026-09-27T23:33:55.976360396Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-btngfc" for cluster "cluster-nightly-4-20-btngfc"
Unexpected error:
    <*fmt.wrapError | 0xc000b4c1e0>: 
    failed to create NodePool np-btngfc: failed starting nodepool creation "np-btngfc" for cluster "cluster-nightly-4-20-btngfc" in resourcegroup="rg-nightly-4-20-dlkdt65gwjpj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-dlkdt65gwjpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-btngfc/nodePools/np-btngfc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-btngfc" for cluster "cluster-nightly-4-20-btngfc"
Unexpected error:
    <*fmt.wrapError | 0xc000b4c1e0>: 
    failed to create NodePool np-btngfc: failed starting nodepool creation "np-btngfc" for cluster "cluster-nightly-4-20-btngfc" in resourcegroup="rg-nightly-4-20-dlkdt65gwjpj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-dlkdt65gwjpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-btngfc/nodePools/np-btngfc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0026392a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pndxf7r8t9rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143",
      "name": "a91e490f-050a-4c07-a58f-af347cb9f143",
      "status": "Failed",
      "startTime": "2026-09-28T00:13:40.174684426Z",
      "endTime": "2026-09-28T00:42:43.628005146Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0026392a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pndxf7r8t9rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143",
      "name": "a91e490f-050a-4c07-a58f-af347cb9f143",
      "status": "Failed",
      "startTime": "2026-09-28T00:13:40.174684426Z",
      "endTime": "2026-09-28T00:42:43.628005146Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000992660>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-nwqdmg89c6p9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "name": "7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:36.690582806Z",
      "endTime": "2026-09-27T23:31:45.971437914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000992660>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-nwqdmg89c6p9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "name": "7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:36.690582806Z",
      "endTime": "2026-09-27T23:31:45.971437914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc001068760>: 
    failed to create HCP cluster ea-listjxmbqs: failed waiting for cluster="ea-listjxmbqs" in resourcegroup="ea-list-hdczj2g88mkl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91bd5de8-aa42-4e7a-b546-48535f13c304
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91bd5de8-aa42-4e7a-b546-48535f13c304",
      "name": "91bd5de8-aa42-4e7a-b546-48535f13c304",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:57.04687797Z",
      "endTime": "2026-09-28T00:28:03.571739169Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc001068760>: 
    failed to create HCP cluster ea-listjxmbqs: failed waiting for cluster="ea-listjxmbqs" in resourcegroup="ea-list-hdczj2g88mkl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91bd5de8-aa42-4e7a-b546-48535f13c304
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91bd5de8-aa42-4e7a-b546-48535f13c304",
      "name": "91bd5de8-aa42-4e7a-b546-48535f13c304",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:57.04687797Z",
      "endTime": "2026-09-28T00:28:03.571739169Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0014c0f60>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/75179e80-a77b-471b-9ea6-7fe700fba9de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/75179e80-a77b-471b-9ea6-7fe700fba9de",
      "name": "75179e80-a77b-471b-9ea6-7fe700fba9de",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:07.551463967Z",
      "endTime": "2026-09-27T23:32:15.972919815Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0014c0f60>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/75179e80-a77b-471b-9ea6-7fe700fba9de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/75179e80-a77b-471b-9ea6-7fe700fba9de",
      "name": "75179e80-a77b-471b-9ea6-7fe700fba9de",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:07.551463967Z",
      "endTime": "2026-09-27T23:32:15.972919815Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc001014fe0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-c85djlsc4gqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a11dc30e-24bb-4e65-a64f-d5c4a10261d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a11dc30e-24bb-4e65-a64f-d5c4a10261d0",
      "name": "a11dc30e-24bb-4e65-a64f-d5c4a10261d0",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:30.190180423Z",
      "endTime": "2026-09-28T00:28:33.57420533Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc001014fe0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-c85djlsc4gqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a11dc30e-24bb-4e65-a64f-d5c4a10261d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a11dc30e-24bb-4e65-a64f-d5c4a10261d0",
      "name": "a11dc30e-24bb-4e65-a64f-d5c4a10261d0",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:30.190180423Z",
      "endTime": "2026-09-28T00:28:33.57420533Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-q2v9b92t5wtw/cluster-nightly-4-21-255l2v should provision
Unexpected error:
    <*fmt.wrapError | 0xc00100e740>: 
    failed to create HCP cluster cluster-nightly-4-21-255l2v: failed waiting for cluster="cluster-nightly-4-21-255l2v" in resourcegroup="rg-nightly-4-21-q2v9b92t5wtw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04aa15fa-a787-49ca-95e4-fb62f01ffcb1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04aa15fa-a787-49ca-95e4-fb62f01ffcb1",
      "name": "04aa15fa-a787-49ca-95e4-fb62f01ffcb1",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:08.296306358Z",
      "endTime": "2026-09-28T00:26:13.568890404Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-q2v9b92t5wtw/cluster-nightly-4-21-255l2v should provision
Unexpected error:
    <*fmt.wrapError | 0xc00100e740>: 
    failed to create HCP cluster cluster-nightly-4-21-255l2v: failed waiting for cluster="cluster-nightly-4-21-255l2v" in resourcegroup="rg-nightly-4-21-q2v9b92t5wtw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04aa15fa-a787-49ca-95e4-fb62f01ffcb1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04aa15fa-a787-49ca-95e4-fb62f01ffcb1",
      "name": "04aa15fa-a787-49ca-95e4-fb62f01ffcb1",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:08.296306358Z",
      "endTime": "2026-09-28T00:26:13.568890404Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000fc6900>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ppfrxnhlq6zz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a75d97c-2b17-4c11-9379-14857f7a504f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a75d97c-2b17-4c11-9379-14857f7a504f",
      "name": "7a75d97c-2b17-4c11-9379-14857f7a504f",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:37.656261821Z",
      "endTime": "2026-09-27T23:31:45.971502523Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000fc6900>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ppfrxnhlq6zz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a75d97c-2b17-4c11-9379-14857f7a504f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a75d97c-2b17-4c11-9379-14857f7a504f",
      "name": "7a75d97c-2b17-4c11-9379-14857f7a504f",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:37.656261821Z",
      "endTime": "2026-09-27T23:31:45.971502523Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00083a9e0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7pdd9b2qtsjh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55bd848b-764d-4e14-887a-5c8e865b8b02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55bd848b-764d-4e14-887a-5c8e865b8b02",
      "name": "55bd848b-764d-4e14-887a-5c8e865b8b02",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:38.240422282Z",
      "endTime": "2026-09-27T23:31:45.97016158Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00083a9e0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7pdd9b2qtsjh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55bd848b-764d-4e14-887a-5c8e865b8b02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55bd848b-764d-4e14-887a-5c8e865b8b02",
      "name": "55bd848b-764d-4e14-887a-5c8e865b8b02",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:38.240422282Z",
      "endTime": "2026-09-27T23:31:45.97016158Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fbjrkv" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000adaa00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-fbjrkv: failed waiting for cluster="cp-ystream-upgrade-4-23-fbjrkv" in resourcegroup="rg-cp-ystream-upgrade-4-23-xs76x2g6dgbl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "name": "2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:39.891778199Z",
      "endTime": "2026-09-28T00:26:43.608164604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fbjrkv" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000adaa00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-fbjrkv: failed waiting for cluster="cp-ystream-upgrade-4-23-fbjrkv" in resourcegroup="rg-cp-ystream-upgrade-4-23-xs76x2g6dgbl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "name": "2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:39.891778199Z",
      "endTime": "2026-09-28T00:26:43.608164604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000866340>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wg62gnrkcq82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "name": "38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:28.308542215Z",
      "endTime": "2026-09-27T23:31:35.969190413Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000866340>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wg62gnrkcq82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "name": "38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:28.308542215Z",
      "endTime": "2026-09-27T23:31:35.969190413Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-4jr4wg" at install version 4.21.0-0.nightly-multi-2026-08-16-064216
Unexpected error:
    <*fmt.wrapError | 0xc000b9e860>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-4jr4wg: failed waiting for cluster="cp-ystream-upgrade-4-22-4jr4wg" in resourcegroup="rg-cp-ystream-upgrade-4-22-8vztd747qdlm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40ab65c1-e843-47e1-be96-78d0820ab888
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40ab65c1-e843-47e1-be96-78d0820ab888",
      "name": "40ab65c1-e843-47e1-be96-78d0820ab888",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.974586451Z",
      "endTime": "2026-09-28T00:26:23.56725042Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-4jr4wg" at install version 4.21.0-0.nightly-multi-2026-08-16-064216
Unexpected error:
    <*fmt.wrapError | 0xc000b9e860>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-4jr4wg: failed waiting for cluster="cp-ystream-upgrade-4-22-4jr4wg" in resourcegroup="rg-cp-ystream-upgrade-4-22-8vztd747qdlm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40ab65c1-e843-47e1-be96-78d0820ab888
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40ab65c1-e843-47e1-be96-78d0820ab888",
      "name": "40ab65c1-e843-47e1-be96-78d0820ab888",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.974586451Z",
      "endTime": "2026-09-28T00:26:23.56725042Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c38ae0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mqcj6sgbzwzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4562cc54-0f79-4e54-9f42-579f7446258f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4562cc54-0f79-4e54-9f42-579f7446258f",
      "name": "4562cc54-0f79-4e54-9f42-579f7446258f",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:27.392724975Z",
      "endTime": "2026-09-27T23:31:35.969501842Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c38ae0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mqcj6sgbzwzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4562cc54-0f79-4e54-9f42-579f7446258f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4562cc54-0f79-4e54-9f42-579f7446258f",
      "name": "4562cc54-0f79-4e54-9f42-579f7446258f",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:27.392724975Z",
      "endTime": "2026-09-27T23:31:35.969501842Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d8ea40>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gtjvj95k94qz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/340b72ab-640f-4ead-8482-0990a3a001a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/340b72ab-640f-4ead-8482-0990a3a001a4",
      "name": "340b72ab-640f-4ead-8482-0990a3a001a4",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:11.801487325Z",
      "endTime": "2026-09-27T23:33:15.975699789Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d8ea40>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gtjvj95k94qz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/340b72ab-640f-4ead-8482-0990a3a001a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/340b72ab-640f-4ead-8482-0990a3a001a4",
      "name": "340b72ab-640f-4ead-8482-0990a3a001a4",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:11.801487325Z",
      "endTime": "2026-09-27T23:33:15.975699789Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a56bc0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-hffn6kx6ns6p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d3c213fe-3996-4b11-8d9a-d57af133e8ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d3c213fe-3996-4b11-8d9a-d57af133e8ac",
      "name": "d3c213fe-3996-4b11-8d9a-d57af133e8ac",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:47.483810554Z",
      "endTime": "2026-09-28T00:27:53.571341129Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a56bc0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-hffn6kx6ns6p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d3c213fe-3996-4b11-8d9a-d57af133e8ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d3c213fe-3996-4b11-8d9a-d57af133e8ac",
      "name": "d3c213fe-3996-4b11-8d9a-d57af133e8ac",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:47.483810554Z",
      "endTime": "2026-09-28T00:27:53.571341129Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000cdc300>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-pj47jh" in resourcegroup="rg-np-skip-minor-pj47jh-jmcx69spqbgl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-pj47jh-jmcx69spqbgl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-pj47jh/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000cdc300>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-pj47jh" in resourcegroup="rg-np-skip-minor-pj47jh-jmcx69spqbgl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-pj47jh-jmcx69spqbgl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-pj47jh/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001014500>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-gdl957wjhnmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61551070-5bea-4c95-8cd2-94223dd885de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61551070-5bea-4c95-8cd2-94223dd885de",
      "name": "61551070-5bea-4c95-8cd2-94223dd885de",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:37.300565328Z",
      "endTime": "2026-09-27T23:31:45.970879971Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001014500>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-gdl957wjhnmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61551070-5bea-4c95-8cd2-94223dd885de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61551070-5bea-4c95-8cd2-94223dd885de",
      "name": "61551070-5bea-4c95-8cd2-94223dd885de",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:37.300565328Z",
      "endTime": "2026-09-27T23:31:45.970879971Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000f96920>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-z49h7x9dg72j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efbc8d98-3e3c-4be8-a742-fceb1317ce9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efbc8d98-3e3c-4be8-a742-fceb1317ce9e",
      "name": "efbc8d98-3e3c-4be8-a742-fceb1317ce9e",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:11.709436376Z",
      "endTime": "2026-09-27T23:34:15.976995944Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000f96920>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-z49h7x9dg72j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efbc8d98-3e3c-4be8-a742-fceb1317ce9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efbc8d98-3e3c-4be8-a742-fceb1317ce9e",
      "name": "efbc8d98-3e3c-4be8-a742-fceb1317ce9e",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:11.709436376Z",
      "endTime": "2026-09-27T23:34:15.976995944Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000c53960>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-klr6qxl7tgzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/721a00dd-f703-40c1-b1bd-8a22d7beaa2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/721a00dd-f703-40c1-b1bd-8a22d7beaa2b",
      "name": "721a00dd-f703-40c1-b1bd-8a22d7beaa2b",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:30.932322501Z",
      "endTime": "2026-09-27T23:32:35.974576236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000c53960>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-klr6qxl7tgzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/721a00dd-f703-40c1-b1bd-8a22d7beaa2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/721a00dd-f703-40c1-b1bd-8a22d7beaa2b",
      "name": "721a00dd-f703-40c1-b1bd-8a22d7beaa2b",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:30.932322501Z",
      "endTime": "2026-09-27T23:32:35.974576236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-j6wfl7rfmftf/cluster-nightly-4-22-nm6bdd should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b18ea0>: 
    failed to create HCP cluster cluster-nightly-4-22-nm6bdd: failed waiting for cluster="cluster-nightly-4-22-nm6bdd" in resourcegroup="rg-nightly-4-22-j6wfl7rfmftf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "name": "30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.981848903Z",
      "endTime": "2026-09-28T00:26:23.602322446Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-j6wfl7rfmftf/cluster-nightly-4-22-nm6bdd should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b18ea0>: 
    failed to create HCP cluster cluster-nightly-4-22-nm6bdd: failed waiting for cluster="cluster-nightly-4-22-nm6bdd" in resourcegroup="rg-nightly-4-22-j6wfl7rfmftf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "name": "30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.981848903Z",
      "endTime": "2026-09-28T00:26:23.602322446Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-mvb2mp2h8jkn/cluster-nightly-5-1-bjw648 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001076960>: 
    failed to create HCP cluster cluster-nightly-5-1-bjw648: failed waiting for cluster="cluster-nightly-5-1-bjw648" in resourcegroup="rg-nightly-5-1-mvb2mp2h8jkn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a050b64c-6ea3-4b3d-bd9c-59727e5486cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a050b64c-6ea3-4b3d-bd9c-59727e5486cc",
      "name": "a050b64c-6ea3-4b3d-bd9c-59727e5486cc",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.688455869Z",
      "endTime": "2026-09-28T00:26:23.566602132Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-mvb2mp2h8jkn/cluster-nightly-5-1-bjw648 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001076960>: 
    failed to create HCP cluster cluster-nightly-5-1-bjw648: failed waiting for cluster="cluster-nightly-5-1-bjw648" in resourcegroup="rg-nightly-5-1-mvb2mp2h8jkn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a050b64c-6ea3-4b3d-bd9c-59727e5486cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a050b64c-6ea3-4b3d-bd9c-59727e5486cc",
      "name": "a050b64c-6ea3-4b3d-bd9c-59727e5486cc",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.688455869Z",
      "endTime": "2026-09-28T00:26:23.566602132Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000feaa80>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-ss6k2z4f5mr9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2dffefc9-9875-40d1-ba0f-58f0a11d1a91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2dffefc9-9875-40d1-ba0f-58f0a11d1a91",
      "name": "2dffefc9-9875-40d1-ba0f-58f0a11d1a91",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:12.795655613Z",
      "endTime": "2026-09-27T23:33:15.9764747Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000feaa80>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-ss6k2z4f5mr9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2dffefc9-9875-40d1-ba0f-58f0a11d1a91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2dffefc9-9875-40d1-ba0f-58f0a11d1a91",
      "name": "2dffefc9-9875-40d1-ba0f-58f0a11d1a91",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:12.795655613Z",
      "endTime": "2026-09-27T23:33:15.9764747Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000b8f440>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-k5f54rh29tw9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2695f42e-c81b-4e4b-95ae-936410e8590b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2695f42e-c81b-4e4b-95ae-936410e8590b",
      "name": "2695f42e-c81b-4e4b-95ae-936410e8590b",
      "status": "Failed",
      "startTime": "2026-09-28T00:19:47.758187128Z",
      "endTime": "2026-09-28T00:48:53.658149566Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000b8f440>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-k5f54rh29tw9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2695f42e-c81b-4e4b-95ae-936410e8590b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2695f42e-c81b-4e4b-95ae-936410e8590b",
      "name": "2695f42e-c81b-4e4b-95ae-936410e8590b",
      "status": "Failed",
      "startTime": "2026-09-28T00:19:47.758187128Z",
      "endTime": "2026-09-28T00:48:53.658149566Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc001088760>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-zxmzb298997v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/92501662-6df7-42a3-aa43-bcd5971c9ba9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/92501662-6df7-42a3-aa43-bcd5971c9ba9",
      "name": "92501662-6df7-42a3-aa43-bcd5971c9ba9",
      "status": "Failed",
      "startTime": "2026-09-27T23:56:24.733972836Z",
      "endTime": "2026-09-28T00:25:33.565873088Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc001088760>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-zxmzb298997v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/92501662-6df7-42a3-aa43-bcd5971c9ba9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/92501662-6df7-42a3-aa43-bcd5971c9ba9",
      "name": "92501662-6df7-42a3-aa43-bcd5971c9ba9",
      "status": "Failed",
      "startTime": "2026-09-27T23:56:24.733972836Z",
      "endTime": "2026-09-28T00:25:33.565873088Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxwdj8sw and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-fct27gcqstbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0007a08f0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxwdj8sw and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-fct27gcqstbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0007a08f0>: 
    storage account log verification failed
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000a90f20>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-nsqwrgcngwlr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ffcb232-74e7-4ea1-baf9-e6e3b272dd4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ffcb232-74e7-4ea1-baf9-e6e3b272dd4d",
      "name": "4ffcb232-74e7-4ea1-baf9-e6e3b272dd4d",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:38.058655228Z",
      "endTime": "2026-09-27T23:31:45.971610274Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000a90f20>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-nsqwrgcngwlr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ffcb232-74e7-4ea1-baf9-e6e3b272dd4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ffcb232-74e7-4ea1-baf9-e6e3b272dd4d",
      "name": "4ffcb232-74e7-4ea1-baf9-e6e3b272dd4d",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:38.058655228Z",
      "endTime": "2026-09-27T23:31:45.971610274Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc001037020>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-nbnflj48n7st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2fc333a9-a61b-4943-8047-7c9a4ad8b0d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2fc333a9-a61b-4943-8047-7c9a4ad8b0d9",
      "name": "2fc333a9-a61b-4943-8047-7c9a4ad8b0d9",
      "status": "Failed",
      "startTime": "2026-09-27T23:56:24.530831437Z",
      "endTime": "2026-09-28T00:25:33.566328828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc001037020>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-nbnflj48n7st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2fc333a9-a61b-4943-8047-7c9a4ad8b0d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2fc333a9-a61b-4943-8047-7c9a4ad8b0d9",
      "name": "2fc333a9-a61b-4943-8047-7c9a4ad8b0d9",
      "status": "Failed",
      "startTime": "2026-09-27T23:56:24.530831437Z",
      "endTime": "2026-09-28T00:25:33.566328828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000b13b60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-6k9kth8pvmf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "name": "eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "status": "Failed",
      "startTime": "2026-09-27T02:13:34.441508798Z",
      "endTime": "2026-09-27T02:32:39.228262555Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000b13b60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-6k9kth8pvmf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "name": "eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "status": "Failed",
      "startTime": "2026-09-27T02:13:34.441508798Z",
      "endTime": "2026-09-27T02:32:39.228262555Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxt4mfpn and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-vg2849dzv4lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0004c67b0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxt4mfpn and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-vg2849dzv4lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0004c67b0>: 
    storage account log verification failed
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc00106a880>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-6snh6pj58f7d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83fb43bb-0f59-41e0-9a1b-8448655aa26b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83fb43bb-0f59-41e0-9a1b-8448655aa26b",
      "name": "83fb43bb-0f59-41e0-9a1b-8448655aa26b",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:39.021853494Z",
      "endTime": "2026-09-27T23:31:39.487220639Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc00106a880>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-6snh6pj58f7d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83fb43bb-0f59-41e0-9a1b-8448655aa26b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83fb43bb-0f59-41e0-9a1b-8448655aa26b",
      "name": "83fb43bb-0f59-41e0-9a1b-8448655aa26b",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:39.021853494Z",
      "endTime": "2026-09-27T23:31:39.487220639Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-rc6w6xrwc9jq/cluster-nightly-4-21-t5fxdv should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c82980>: 
    failed to create HCP cluster cluster-nightly-4-21-t5fxdv: failed waiting for cluster="cluster-nightly-4-21-t5fxdv" in resourcegroup="rg-nightly-4-21-rc6w6xrwc9jq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a7b3551-f75b-40ce-ab2a-fa119dc496ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a7b3551-f75b-40ce-ab2a-fa119dc496ea",
      "name": "3a7b3551-f75b-40ce-ab2a-fa119dc496ea",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:27.933440104Z",
      "endTime": "2026-09-28T00:26:27.662365112Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-rc6w6xrwc9jq/cluster-nightly-4-21-t5fxdv should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c82980>: 
    failed to create HCP cluster cluster-nightly-4-21-t5fxdv: failed waiting for cluster="cluster-nightly-4-21-t5fxdv" in resourcegroup="rg-nightly-4-21-rc6w6xrwc9jq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a7b3551-f75b-40ce-ab2a-fa119dc496ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a7b3551-f75b-40ce-ab2a-fa119dc496ea",
      "name": "3a7b3551-f75b-40ce-ab2a-fa119dc496ea",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:27.933440104Z",
      "endTime": "2026-09-28T00:26:27.662365112Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000af6ac0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-wvkkld894p88" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fcfc5cb-5f34-438a-85db-f4c1fec4ebbc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fcfc5cb-5f34-438a-85db-f4c1fec4ebbc",
      "name": "9fcfc5cb-5f34-438a-85db-f4c1fec4ebbc",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:22.544216392Z",
      "endTime": "2026-09-27T23:34:29.497556122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000af6ac0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-wvkkld894p88" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fcfc5cb-5f34-438a-85db-f4c1fec4ebbc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fcfc5cb-5f34-438a-85db-f4c1fec4ebbc",
      "name": "9fcfc5cb-5f34-438a-85db-f4c1fec4ebbc",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:22.544216392Z",
      "endTime": "2026-09-27T23:34:29.497556122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc001066e60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-vfb8zbnsnff8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8171d8b8-0ee8-41e1-9596-bb419f76fa66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8171d8b8-0ee8-41e1-9596-bb419f76fa66",
      "name": "8171d8b8-0ee8-41e1-9596-bb419f76fa66",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:39.577645259Z",
      "endTime": "2026-09-27T23:31:39.48702037Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc001066e60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-vfb8zbnsnff8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8171d8b8-0ee8-41e1-9596-bb419f76fa66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8171d8b8-0ee8-41e1-9596-bb419f76fa66",
      "name": "8171d8b8-0ee8-41e1-9596-bb419f76fa66",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:39.577645259Z",
      "endTime": "2026-09-27T23:31:39.48702037Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009236a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-nz9q7cgh9pms" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/57404065-d4e5-4294-9f65-1130a8ff568d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/57404065-d4e5-4294-9f65-1130a8ff568d",
      "name": "57404065-d4e5-4294-9f65-1130a8ff568d",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:14.225943204Z",
      "endTime": "2026-09-28T00:27:17.671713535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009236a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-nz9q7cgh9pms" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/57404065-d4e5-4294-9f65-1130a8ff568d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/57404065-d4e5-4294-9f65-1130a8ff568d",
      "name": "57404065-d4e5-4294-9f65-1130a8ff568d",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:14.225943204Z",
      "endTime": "2026-09-28T00:27:17.671713535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bbafa0>: 
    failed to create HCP cluster np-dg-minor-ljmn55: failed waiting for cluster="np-dg-minor-ljmn55" in resourcegroup="rg-np-dg-minor-ljmn55-vsj9pfc6qn9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43d9debf-fbb9-4fcf-9b55-9ba4bb51fc3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43d9debf-fbb9-4fcf-9b55-9ba4bb51fc3b",
      "name": "43d9debf-fbb9-4fcf-9b55-9ba4bb51fc3b",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:45.433531245Z",
      "endTime": "2026-09-28T00:28:47.692103938Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bbafa0>: 
    failed to create HCP cluster np-dg-minor-ljmn55: failed waiting for cluster="np-dg-minor-ljmn55" in resourcegroup="rg-np-dg-minor-ljmn55-vsj9pfc6qn9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43d9debf-fbb9-4fcf-9b55-9ba4bb51fc3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43d9debf-fbb9-4fcf-9b55-9ba4bb51fc3b",
      "name": "43d9debf-fbb9-4fcf-9b55-9ba4bb51fc3b",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:45.433531245Z",
      "endTime": "2026-09-28T00:28:47.692103938Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-kxcpclphszg4/cluster-nightly-4-22-7c2dsj should provision
Unexpected error:
    <*fmt.wrapError | 0xc00088c6e0>: 
    failed to create HCP cluster cluster-nightly-4-22-7c2dsj: failed waiting for cluster="cluster-nightly-4-22-7c2dsj" in resourcegroup="rg-nightly-4-22-kxcpclphszg4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58004c12-b9b9-4d95-8449-fdf773362b87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58004c12-b9b9-4d95-8449-fdf773362b87",
      "name": "58004c12-b9b9-4d95-8449-fdf773362b87",
      "status": "Failed",
      "startTime": "2026-09-27T23:56:56.262478975Z",
      "endTime": "2026-09-28T00:25:57.660643706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-kxcpclphszg4/cluster-nightly-4-22-7c2dsj should provision
Unexpected error:
    <*fmt.wrapError | 0xc00088c6e0>: 
    failed to create HCP cluster cluster-nightly-4-22-7c2dsj: failed waiting for cluster="cluster-nightly-4-22-7c2dsj" in resourcegroup="rg-nightly-4-22-kxcpclphszg4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58004c12-b9b9-4d95-8449-fdf773362b87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58004c12-b9b9-4d95-8449-fdf773362b87",
      "name": "58004c12-b9b9-4d95-8449-fdf773362b87",
      "status": "Failed",
      "startTime": "2026-09-27T23:56:56.262478975Z",
      "endTime": "2026-09-28T00:25:57.660643706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000656de0>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-slrthwkjtv9h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6ad81add-2f55-4337-958c-f7114bdff1ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6ad81add-2f55-4337-958c-f7114bdff1ed",
      "name": "6ad81add-2f55-4337-958c-f7114bdff1ed",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:43.110507621Z",
      "endTime": "2026-09-27T23:31:49.489110364Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000656de0>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-slrthwkjtv9h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6ad81add-2f55-4337-958c-f7114bdff1ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6ad81add-2f55-4337-958c-f7114bdff1ed",
      "name": "6ad81add-2f55-4337-958c-f7114bdff1ed",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:43.110507621Z",
      "endTime": "2026-09-27T23:31:49.489110364Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-h77pz9 with version 4.21.0-0.nightly-multi-2026-08-16-064216
Unexpected error:
    <*fmt.wrapError | 0xc00138a820>: 
    failed to create HCP cluster np-version-upgrade-cluster-h77pz9: failed waiting for cluster="np-version-upgrade-cluster-h77pz9" in resourcegroup="rg-np-version-upgrade-h77pz9-6r8s69c7447k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3ba77d7f-abbf-4320-bc76-508bf6a329cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3ba77d7f-abbf-4320-bc76-508bf6a329cc",
      "name": "3ba77d7f-abbf-4320-bc76-508bf6a329cc",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:32.168616266Z",
      "endTime": "2026-09-28T00:27:37.693340161Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-h77pz9 with version 4.21.0-0.nightly-multi-2026-08-16-064216
Unexpected error:
    <*fmt.wrapError | 0xc00138a820>: 
    failed to create HCP cluster np-version-upgrade-cluster-h77pz9: failed waiting for cluster="np-version-upgrade-cluster-h77pz9" in resourcegroup="rg-np-version-upgrade-h77pz9-6r8s69c7447k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3ba77d7f-abbf-4320-bc76-508bf6a329cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3ba77d7f-abbf-4320-bc76-508bf6a329cc",
      "name": "3ba77d7f-abbf-4320-bc76-508bf6a329cc",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:32.168616266Z",
      "endTime": "2026-09-28T00:27:37.693340161Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000a0aee0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-554wk26tw8h2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/10be9e51-c88b-4df0-a061-31b074ee852b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/10be9e51-c88b-4df0-a061-31b074ee852b",
      "name": "10be9e51-c88b-4df0-a061-31b074ee852b",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:25.519573959Z",
      "endTime": "2026-09-27T23:33:29.494256598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000a0aee0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-554wk26tw8h2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/10be9e51-c88b-4df0-a061-31b074ee852b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/10be9e51-c88b-4df0-a061-31b074ee852b",
      "name": "10be9e51-c88b-4df0-a061-31b074ee852b",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:25.519573959Z",
      "endTime": "2026-09-27T23:33:29.494256598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d9e5c0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lzzgsqgdxjgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/21bbf8c9-7a2e-43a3-bf73-bb1271c743ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/21bbf8c9-7a2e-43a3-bf73-bb1271c743ab",
      "name": "21bbf8c9-7a2e-43a3-bf73-bb1271c743ab",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:09.498879782Z",
      "endTime": "2026-09-28T00:27:17.67319034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d9e5c0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lzzgsqgdxjgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/21bbf8c9-7a2e-43a3-bf73-bb1271c743ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/21bbf8c9-7a2e-43a3-bf73-bb1271c743ab",
      "name": "21bbf8c9-7a2e-43a3-bf73-bb1271c743ab",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:09.498879782Z",
      "endTime": "2026-09-28T00:27:17.67319034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001280460>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-vqg75gmd6vnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e87911dd-0e87-4c8b-bdb2-aa982e1192f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e87911dd-0e87-4c8b-bdb2-aa982e1192f2",
      "name": "e87911dd-0e87-4c8b-bdb2-aa982e1192f2",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.841093241Z",
      "endTime": "2026-09-27T23:31:59.489031363Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001280460>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-vqg75gmd6vnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e87911dd-0e87-4c8b-bdb2-aa982e1192f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e87911dd-0e87-4c8b-bdb2-aa982e1192f2",
      "name": "e87911dd-0e87-4c8b-bdb2-aa982e1192f2",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.841093241Z",
      "endTime": "2026-09-27T23:31:59.489031363Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f1ea60>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-ptqx8bknktzp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4f2ab8e-cc8b-461a-8624-5179be54eabc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4f2ab8e-cc8b-461a-8624-5179be54eabc",
      "name": "d4f2ab8e-cc8b-461a-8624-5179be54eabc",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.747714732Z",
      "endTime": "2026-09-27T23:31:59.490551228Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f1ea60>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-ptqx8bknktzp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4f2ab8e-cc8b-461a-8624-5179be54eabc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4f2ab8e-cc8b-461a-8624-5179be54eabc",
      "name": "d4f2ab8e-cc8b-461a-8624-5179be54eabc",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.747714732Z",
      "endTime": "2026-09-27T23:31:59.490551228Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f40d60>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-shvcm52drf9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01b938cc-52a2-4887-89f3-2d3cad2b72c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01b938cc-52a2-4887-89f3-2d3cad2b72c3",
      "name": "01b938cc-52a2-4887-89f3-2d3cad2b72c3",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:07.744422468Z",
      "endTime": "2026-09-28T00:27:07.672033243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f40d60>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-shvcm52drf9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01b938cc-52a2-4887-89f3-2d3cad2b72c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01b938cc-52a2-4887-89f3-2d3cad2b72c3",
      "name": "01b938cc-52a2-4887-89f3-2d3cad2b72c3",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:07.744422468Z",
      "endTime": "2026-09-28T00:27:07.672033243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011eadb0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011eadb0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b98720>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vh7rhmvj6b82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82682c87-0613-4528-bbdc-b0b5471d89e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82682c87-0613-4528-bbdc-b0b5471d89e5",
      "name": "82682c87-0613-4528-bbdc-b0b5471d89e5",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:47.113470008Z",
      "endTime": "2026-09-28T00:28:47.692572721Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b98720>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vh7rhmvj6b82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82682c87-0613-4528-bbdc-b0b5471d89e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82682c87-0613-4528-bbdc-b0b5471d89e5",
      "name": "82682c87-0613-4528-bbdc-b0b5471d89e5",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:47.113470008Z",
      "endTime": "2026-09-28T00:28:47.692572721Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000908760>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-rdnft58nhhhs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6e6da606-72ef-4906-b49f-2e0d0f42ec52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6e6da606-72ef-4906-b49f-2e0d0f42ec52",
      "name": "6e6da606-72ef-4906-b49f-2e0d0f42ec52",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:14.609473044Z",
      "endTime": "2026-09-28T00:27:17.672444419Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000908760>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-rdnft58nhhhs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6e6da606-72ef-4906-b49f-2e0d0f42ec52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6e6da606-72ef-4906-b49f-2e0d0f42ec52",
      "name": "6e6da606-72ef-4906-b49f-2e0d0f42ec52",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:14.609473044Z",
      "endTime": "2026-09-28T00:27:17.672444419Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f66c00>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-pbjsp2slkg85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72800f77-818e-4db0-9b79-7269b7c09db1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72800f77-818e-4db0-9b79-7269b7c09db1",
      "name": "72800f77-818e-4db0-9b79-7269b7c09db1",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.678838745Z",
      "endTime": "2026-09-27T23:31:59.489386628Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f66c00>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-pbjsp2slkg85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72800f77-818e-4db0-9b79-7269b7c09db1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/72800f77-818e-4db0-9b79-7269b7c09db1",
      "name": "72800f77-818e-4db0-9b79-7269b7c09db1",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.678838745Z",
      "endTime": "2026-09-27T23:31:59.489386628Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-ls987j" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a12660>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-ls987j: failed waiting for cluster="cp-ystream-upgrade-4-23-ls987j" in resourcegroup="rg-cp-ystream-upgrade-4-23-hstckbwcrnkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a32dc68b-9126-427b-be57-cfe4cc635070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a32dc68b-9126-427b-be57-cfe4cc635070",
      "name": "a32dc68b-9126-427b-be57-cfe4cc635070",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:18.507273215Z",
      "endTime": "2026-09-28T00:27:27.673315974Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-ls987j" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a12660>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-ls987j: failed waiting for cluster="cp-ystream-upgrade-4-23-ls987j" in resourcegroup="rg-cp-ystream-upgrade-4-23-hstckbwcrnkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a32dc68b-9126-427b-be57-cfe4cc635070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a32dc68b-9126-427b-be57-cfe4cc635070",
      "name": "a32dc68b-9126-427b-be57-cfe4cc635070",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:18.507273215Z",
      "endTime": "2026-09-28T00:27:27.673315974Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001058aa0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-vd4sq7v444zm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b50c03-8593-4772-9894-b52e190e1fff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b50c03-8593-4772-9894-b52e190e1fff",
      "name": "d5b50c03-8593-4772-9894-b52e190e1fff",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:49.532790064Z",
      "endTime": "2026-09-27T23:31:49.489314628Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001058aa0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-vd4sq7v444zm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b50c03-8593-4772-9894-b52e190e1fff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b50c03-8593-4772-9894-b52e190e1fff",
      "name": "d5b50c03-8593-4772-9894-b52e190e1fff",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:49.532790064Z",
      "endTime": "2026-09-27T23:31:49.489314628Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000f66900>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-rt6qgw6rjndv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f1db31d-2da3-451a-bf69-058f9201d45c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f1db31d-2da3-451a-bf69-058f9201d45c",
      "name": "7f1db31d-2da3-451a-bf69-058f9201d45c",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:18.118119937Z",
      "endTime": "2026-09-28T00:27:27.673012152Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000f66900>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-rt6qgw6rjndv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f1db31d-2da3-451a-bf69-058f9201d45c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f1db31d-2da3-451a-bf69-058f9201d45c",
      "name": "7f1db31d-2da3-451a-bf69-058f9201d45c",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:18.118119937Z",
      "endTime": "2026-09-28T00:27:27.673012152Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-lhwzsr" at install version 4.21.0-0.nightly-multi-2026-08-16-064216
Unexpected error:
    <*fmt.wrapError | 0xc000eb8de0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-lhwzsr: failed waiting for cluster="cp-ystream-upgrade-4-22-lhwzsr" in resourcegroup="rg-cp-ystream-upgrade-4-22-728lztvgsmqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f3738ec-b0f1-4f12-8cce-11f2283bdc71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f3738ec-b0f1-4f12-8cce-11f2283bdc71",
      "name": "8f3738ec-b0f1-4f12-8cce-11f2283bdc71",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:37.862054451Z",
      "endTime": "2026-09-28T00:26:37.695691601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-lhwzsr" at install version 4.21.0-0.nightly-multi-2026-08-16-064216
Unexpected error:
    <*fmt.wrapError | 0xc000eb8de0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-lhwzsr: failed waiting for cluster="cp-ystream-upgrade-4-22-lhwzsr" in resourcegroup="rg-cp-ystream-upgrade-4-22-728lztvgsmqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f3738ec-b0f1-4f12-8cce-11f2283bdc71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f3738ec-b0f1-4f12-8cce-11f2283bdc71",
      "name": "8f3738ec-b0f1-4f12-8cce-11f2283bdc71",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:37.862054451Z",
      "endTime": "2026-09-28T00:26:37.695691601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-smz8ccw67j95/cluster-nightly-5-1-n44gxt should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a58f80>: 
    failed to create HCP cluster cluster-nightly-5-1-n44gxt: failed waiting for cluster="cluster-nightly-5-1-n44gxt" in resourcegroup="rg-nightly-5-1-smz8ccw67j95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f423c052-75d2-4bae-b4ca-4718d3554963
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f423c052-75d2-4bae-b4ca-4718d3554963",
      "name": "f423c052-75d2-4bae-b4ca-4718d3554963",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:20.81324454Z",
      "endTime": "2026-09-28T00:26:27.66160026Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-smz8ccw67j95/cluster-nightly-5-1-n44gxt should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a58f80>: 
    failed to create HCP cluster cluster-nightly-5-1-n44gxt: failed waiting for cluster="cluster-nightly-5-1-n44gxt" in resourcegroup="rg-nightly-5-1-smz8ccw67j95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f423c052-75d2-4bae-b4ca-4718d3554963
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f423c052-75d2-4bae-b4ca-4718d3554963",
      "name": "f423c052-75d2-4bae-b4ca-4718d3554963",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:20.81324454Z",
      "endTime": "2026-09-28T00:26:27.66160026Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-kq44rx at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a482e0>: 
    failed to create HCP cluster np-skip-minor-kq44rx: failed waiting for cluster="np-skip-minor-kq44rx" in resourcegroup="rg-np-skip-minor-kq44rx-wrs7cdgqmf9c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48",
      "name": "123e618f-af28-40f6-94b5-7da4afca4a48",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:09.384523535Z",
      "endTime": "2026-09-28T00:28:17.690643945Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-kq44rx at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a482e0>: 
    failed to create HCP cluster np-skip-minor-kq44rx: failed waiting for cluster="np-skip-minor-kq44rx" in resourcegroup="rg-np-skip-minor-kq44rx-wrs7cdgqmf9c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48",
      "name": "123e618f-af28-40f6-94b5-7da4afca4a48",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:09.384523535Z",
      "endTime": "2026-09-28T00:28:17.690643945Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000caaca0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-mm5xdfcrnxcg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f52e0df8-8a9a-4d7c-a040-82d256d107d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f52e0df8-8a9a-4d7c-a040-82d256d107d9",
      "name": "f52e0df8-8a9a-4d7c-a040-82d256d107d9",
      "status": "Failed",
      "startTime": "2026-09-28T00:13:43.929725977Z",
      "endTime": "2026-09-28T00:42:50.716503716Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000caaca0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-mm5xdfcrnxcg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f52e0df8-8a9a-4d7c-a040-82d256d107d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f52e0df8-8a9a-4d7c-a040-82d256d107d9",
      "name": "f52e0df8-8a9a-4d7c-a040-82d256d107d9",
      "status": "Failed",
      "startTime": "2026-09-28T00:13:43.929725977Z",
      "endTime": "2026-09-28T00:42:50.716503716Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000b67200>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-697wjx7zzzdm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43b94205-fe8d-47b3-a4a4-7e1266efc4c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43b94205-fe8d-47b3-a4a4-7e1266efc4c0",
      "name": "43b94205-fe8d-47b3-a4a4-7e1266efc4c0",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:15.695573565Z",
      "endTime": "2026-09-28T00:27:17.672891195Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000b67200>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-697wjx7zzzdm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43b94205-fe8d-47b3-a4a4-7e1266efc4c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/43b94205-fe8d-47b3-a4a4-7e1266efc4c0",
      "name": "43b94205-fe8d-47b3-a4a4-7e1266efc4c0",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:15.695573565Z",
      "endTime": "2026-09-28T00:27:17.672891195Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000db4b20>: 
    failed to create HCP cluster ea-listdnk44k: failed waiting for cluster="ea-listdnk44k" in resourcegroup="ea-list-t5csnm9zbqrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0c6958a-5bab-435e-948c-a6226195419e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0c6958a-5bab-435e-948c-a6226195419e",
      "name": "e0c6958a-5bab-435e-948c-a6226195419e",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:08.01833551Z",
      "endTime": "2026-09-28T00:27:17.672737432Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000db4b20>: 
    failed to create HCP cluster ea-listdnk44k: failed waiting for cluster="ea-listdnk44k" in resourcegroup="ea-list-t5csnm9zbqrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0c6958a-5bab-435e-948c-a6226195419e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0c6958a-5bab-435e-948c-a6226195419e",
      "name": "e0c6958a-5bab-435e-948c-a6226195419e",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:08.01833551Z",
      "endTime": "2026-09-28T00:27:17.672737432Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000fcc520>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-47d827vvrw49" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb385e97-6ba2-4c01-9779-4cf02866139b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb385e97-6ba2-4c01-9779-4cf02866139b",
      "name": "eb385e97-6ba2-4c01-9779-4cf02866139b",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:06.639027634Z",
      "endTime": "2026-09-27T23:34:09.49706636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000fcc520>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-47d827vvrw49" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb385e97-6ba2-4c01-9779-4cf02866139b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb385e97-6ba2-4c01-9779-4cf02866139b",
      "name": "eb385e97-6ba2-4c01-9779-4cf02866139b",
      "status": "Failed",
      "startTime": "2026-09-27T23:05:06.639027634Z",
      "endTime": "2026-09-27T23:34:09.49706636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7lc2wg" for cluster "cluster-nightly-4-20-7lc2wg"
Unexpected error:
    <*fmt.wrapError | 0xc000de6640>: 
    failed to create NodePool np-7lc2wg: failed starting nodepool creation "np-7lc2wg" for cluster "cluster-nightly-4-20-7lc2wg" in resourcegroup="rg-nightly-4-20-7g2xnmzkrznn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-7g2xnmzkrznn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-7lc2wg/nodePools/np-7lc2wg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7lc2wg" for cluster "cluster-nightly-4-20-7lc2wg"
Unexpected error:
    <*fmt.wrapError | 0xc000de6640>: 
    failed to create NodePool np-7lc2wg: failed starting nodepool creation "np-7lc2wg" for cluster "cluster-nightly-4-20-7lc2wg" in resourcegroup="rg-nightly-4-20-7g2xnmzkrznn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-7g2xnmzkrznn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-7lc2wg/nodePools/np-7lc2wg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000af4a40>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z9xpg6dx44vh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb3bb885-d121-4d31-94cb-2c2e50676335
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb3bb885-d121-4d31-94cb-2c2e50676335",
      "name": "eb3bb885-d121-4d31-94cb-2c2e50676335",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:43.933442761Z",
      "endTime": "2026-09-27T23:31:49.489929931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000af4a40>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z9xpg6dx44vh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb3bb885-d121-4d31-94cb-2c2e50676335
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb3bb885-d121-4d31-94cb-2c2e50676335",
      "name": "eb3bb885-d121-4d31-94cb-2c2e50676335",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:43.933442761Z",
      "endTime": "2026-09-27T23:31:49.489929931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0016b0ae0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fvm5hjf9trrw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd4fad33-ebbb-47e5-94e2-f6a999a393f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd4fad33-ebbb-47e5-94e2-f6a999a393f0",
      "name": "bd4fad33-ebbb-47e5-94e2-f6a999a393f0",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:13.954217314Z",
      "endTime": "2026-09-28T00:27:17.671690658Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0016b0ae0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fvm5hjf9trrw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd4fad33-ebbb-47e5-94e2-f6a999a393f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd4fad33-ebbb-47e5-94e2-f6a999a393f0",
      "name": "bd4fad33-ebbb-47e5-94e2-f6a999a393f0",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:13.954217314Z",
      "endTime": "2026-09-28T00:27:17.671690658Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0000578a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-xq6nmlhdcd84" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc8edfd8-8b25-47bc-9569-bf80fa4decac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc8edfd8-8b25-47bc-9569-bf80fa4decac",
      "name": "bc8edfd8-8b25-47bc-9569-bf80fa4decac",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.158928262Z",
      "endTime": "2026-09-27T23:31:59.489753067Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0000578a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-xq6nmlhdcd84" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc8edfd8-8b25-47bc-9569-bf80fa4decac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc8edfd8-8b25-47bc-9569-bf80fa4decac",
      "name": "bc8edfd8-8b25-47bc-9569-bf80fa4decac",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.158928262Z",
      "endTime": "2026-09-27T23:31:59.489753067Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00075dac0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bk8w68gw94vn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d275dc6f-a5af-4a23-9d0b-63cfeda20389
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d275dc6f-a5af-4a23-9d0b-63cfeda20389",
      "name": "d275dc6f-a5af-4a23-9d0b-63cfeda20389",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:45.557708779Z",
      "endTime": "2026-09-27T23:32:49.493110089Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00075dac0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bk8w68gw94vn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d275dc6f-a5af-4a23-9d0b-63cfeda20389
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d275dc6f-a5af-4a23-9d0b-63cfeda20389",
      "name": "d275dc6f-a5af-4a23-9d0b-63cfeda20389",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:45.557708779Z",
      "endTime": "2026-09-27T23:32:49.493110089Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fd09a0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-twf8pdtc8q4r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d537e3b3-f194-4f0f-8c18-2922a0c2c1cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d537e3b3-f194-4f0f-8c18-2922a0c2c1cb",
      "name": "d537e3b3-f194-4f0f-8c18-2922a0c2c1cb",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:49.901495557Z",
      "endTime": "2026-09-27T23:31:49.488818493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fd09a0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-twf8pdtc8q4r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d537e3b3-f194-4f0f-8c18-2922a0c2c1cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d537e3b3-f194-4f0f-8c18-2922a0c2c1cb",
      "name": "d537e3b3-f194-4f0f-8c18-2922a0c2c1cb",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:49.901495557Z",
      "endTime": "2026-09-27T23:31:49.488818493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001257740>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001257740>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000733e40>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-85nsqn7c57p8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d1c34fa-c57c-4cba-b433-80703e25ab62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d1c34fa-c57c-4cba-b433-80703e25ab62",
      "name": "4d1c34fa-c57c-4cba-b433-80703e25ab62",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.077800351Z",
      "endTime": "2026-09-27T23:31:59.49007937Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000733e40>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-85nsqn7c57p8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d1c34fa-c57c-4cba-b433-80703e25ab62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d1c34fa-c57c-4cba-b433-80703e25ab62",
      "name": "4d1c34fa-c57c-4cba-b433-80703e25ab62",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.077800351Z",
      "endTime": "2026-09-27T23:31:59.49007937Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004df1e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-7fp82sngxtbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7d3c6b5b-9c7f-4734-b248-2bfbc789fea0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7d3c6b5b-9c7f-4734-b248-2bfbc789fea0",
      "name": "7d3c6b5b-9c7f-4734-b248-2bfbc789fea0",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:14.235161565Z",
      "endTime": "2026-09-28T00:27:17.673483272Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004df1e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-7fp82sngxtbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7d3c6b5b-9c7f-4734-b248-2bfbc789fea0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7d3c6b5b-9c7f-4734-b248-2bfbc789fea0",
      "name": "7d3c6b5b-9c7f-4734-b248-2bfbc789fea0",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:14.235161565Z",
      "endTime": "2026-09-28T00:27:17.673483272Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000e02800>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ss8z8mq428kv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "name": "5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.434225001Z",
      "endTime": "2026-09-27T23:31:59.489114255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000e02800>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ss8z8mq428kv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "name": "5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.434225001Z",
      "endTime": "2026-09-27T23:31:59.489114255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000816cc0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-pbxqtq9q975p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/584ba00c-67ec-4958-af2b-1eadce8027d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/584ba00c-67ec-4958-af2b-1eadce8027d4",
      "name": "584ba00c-67ec-4958-af2b-1eadce8027d4",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:16.758777697Z",
      "endTime": "2026-09-28T00:27:17.672510334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000816cc0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-pbxqtq9q975p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/584ba00c-67ec-4958-af2b-1eadce8027d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/584ba00c-67ec-4958-af2b-1eadce8027d4",
      "name": "584ba00c-67ec-4958-af2b-1eadce8027d4",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:16.758777697Z",
      "endTime": "2026-09-28T00:27:17.672510334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-62gr7l" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000cab380>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-62gr7l: failed waiting for cluster="cp-ystream-upgrade-5-0-62gr7l" in resourcegroup="rg-cp-ystream-upgrade-5-0-cswjlcdrpdmb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/feaf6284-b05a-47b3-9cdc-9b544c41a1ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/feaf6284-b05a-47b3-9cdc-9b544c41a1ca",
      "name": "feaf6284-b05a-47b3-9cdc-9b544c41a1ca",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:16.133688106Z",
      "endTime": "2026-09-28T00:27:17.673536863Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-62gr7l" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000cab380>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-62gr7l: failed waiting for cluster="cp-ystream-upgrade-5-0-62gr7l" in resourcegroup="rg-cp-ystream-upgrade-5-0-cswjlcdrpdmb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/feaf6284-b05a-47b3-9cdc-9b544c41a1ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/feaf6284-b05a-47b3-9cdc-9b544c41a1ca",
      "name": "feaf6284-b05a-47b3-9cdc-9b544c41a1ca",
      "status": "Failed",
      "startTime": "2026-09-27T23:58:16.133688106Z",
      "endTime": "2026-09-28T00:27:17.673536863Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010cc6e0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kpx28dg62cnt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a15c63c7-6d52-4412-81db-0529880a3e86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a15c63c7-6d52-4412-81db-0529880a3e86",
      "name": "a15c63c7-6d52-4412-81db-0529880a3e86",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:51.012142821Z",
      "endTime": "2026-09-27T23:31:59.490175937Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010cc6e0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kpx28dg62cnt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a15c63c7-6d52-4412-81db-0529880a3e86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a15c63c7-6d52-4412-81db-0529880a3e86",
      "name": "a15c63c7-6d52-4412-81db-0529880a3e86",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:51.012142821Z",
      "endTime": "2026-09-27T23:31:59.490175937Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc000fde840>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/449c12fa-ccaf-4ac5-ac25-394845c24b25
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/449c12fa-ccaf-4ac5-ac25-394845c24b25",
      "name": "449c12fa-ccaf-4ac5-ac25-394845c24b25",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:30.661590969Z",
      "endTime": "2026-09-27T23:32:39.492855664Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc000fde840>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/449c12fa-ccaf-4ac5-ac25-394845c24b25
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/449c12fa-ccaf-4ac5-ac25-394845c24b25",
      "name": "449c12fa-ccaf-4ac5-ac25-394845c24b25",
      "status": "Failed",
      "startTime": "2026-09-27T23:03:30.661590969Z",
      "endTime": "2026-09-27T23:32:39.492855664Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00062c960>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-64wkhbmzb7g8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3fd835f-0f6e-4edc-8259-f4ba7eb616f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3fd835f-0f6e-4edc-8259-f4ba7eb616f9",
      "name": "a3fd835f-0f6e-4edc-8259-f4ba7eb616f9",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:10.792622349Z",
      "endTime": "2026-09-27T23:33:19.494656442Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00062c960>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-64wkhbmzb7g8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3fd835f-0f6e-4edc-8259-f4ba7eb616f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3fd835f-0f6e-4edc-8259-f4ba7eb616f9",
      "name": "a3fd835f-0f6e-4edc-8259-f4ba7eb616f9",
      "status": "Failed",
      "startTime": "2026-09-27T23:04:10.792622349Z",
      "endTime": "2026-09-27T23:33:19.494656442Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000c9ca20>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-r8slnk5nqrjs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7fb1b7c5-3455-4ff7-a7c1-0a6068710299
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7fb1b7c5-3455-4ff7-a7c1-0a6068710299",
      "name": "7fb1b7c5-3455-4ff7-a7c1-0a6068710299",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:51.892589675Z",
      "endTime": "2026-09-27T23:31:59.489602625Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000c9ca20>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-r8slnk5nqrjs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7fb1b7c5-3455-4ff7-a7c1-0a6068710299
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7fb1b7c5-3455-4ff7-a7c1-0a6068710299",
      "name": "7fb1b7c5-3455-4ff7-a7c1-0a6068710299",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:51.892589675Z",
      "endTime": "2026-09-27T23:31:59.489602625Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000996660>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-z624hhq7js4c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "name": "c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "status": "Failed",
      "startTime": "2026-09-27T02:15:40.256886606Z",
      "endTime": "2026-09-27T02:34:46.354762492Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000996660>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-z624hhq7js4c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "name": "c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "status": "Failed",
      "startTime": "2026-09-27T02:15:40.256886606Z",
      "endTime": "2026-09-27T02:34:46.354762492Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred