fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs"
Unexpected error:
    <*fmt.wrapError | 0xc000f86d20>: 
    failed to create NodePool np-m8jhjs: failed waiting for nodepool="np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs" in resourcegroup="rg-candidate-4-23-f6rffb7vgk7k" to finish creating: GET https://rp.j5063808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e",
      "name": "451c3970-ee1e-443a-8418-d00eafee945e",
      "status": "Failed",
      "startTime": "2026-09-29T00:03:14.031479379Z",
      "endTime": "2026-09-29T00:22:18.567634171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs"
Unexpected error:
    <*fmt.wrapError | 0xc000f86d20>: 
    failed to create NodePool np-m8jhjs: failed waiting for nodepool="np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs" in resourcegroup="rg-candidate-4-23-f6rffb7vgk7k" to finish creating: GET https://rp.j5063808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e",
      "name": "451c3970-ee1e-443a-8418-d00eafee945e",
      "status": "Failed",
      "startTime": "2026-09-29T00:03:14.031479379Z",
      "endTime": "2026-09-29T00:22:18.567634171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T00:14:18Z
  Ended:       2026-09-29T00:26:21Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4504064-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ff7440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-rdwv792csbm5" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "name": "bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "status": "Failed",
      "startTime": "2026-09-28T23:36:16.368959757Z",
      "endTime": "2026-09-28T23:55:24.990728156Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/virtualMachines/b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-d9rrr: Failed: availabilitysets failed to create or update. err: failed to get existing resource fips-enabled-rdwv792csbm5--managed/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/availabilitySets/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '1' seconds a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ff7440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-rdwv792csbm5" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "name": "bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "status": "Failed",
      "startTime": "2026-09-28T23:36:16.368959757Z",
      "endTime": "2026-09-28T23:55:24.990728156Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/virtualMachines/b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-d9rrr: Failed: availabilitysets failed to create or update. err: failed to get existing resource fips-enabled-rdwv792csbm5--managed/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/availabilitySets/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '1' seconds a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000687000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-vdlktwxnm8vh" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fb52b81-388d-4086-8055-43b274743b11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fb52b81-388d-4086-8055-43b274743b11",
      "name": "9fb52b81-388d-4086-8055-43b274743b11",
      "status": "Failed",
      "startTime": "2026-09-28T23:41:09.989503835Z",
      "endTime": "2026-09-28T23:41:15.954914297Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000687000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-vdlktwxnm8vh" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fb52b81-388d-4086-8055-43b274743b11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fb52b81-388d-4086-8055-43b274743b11",
      "name": "9fb52b81-388d-4086-8055-43b274743b11",
      "status": "Failed",
      "startTime": "2026-09-28T23:41:09.989503835Z",
      "endTime": "2026-09-28T23:41:15.954914297Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000b234e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-nl2nr6" in resourcegroup="rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e",
      "name": "af93dcf2-82e0-4298-856d-f4846917b52e",
      "status": "Failed",
      "startTime": "2026-09-28T23:46:23.744577199Z",
      "endTime": "2026-09-29T00:05:30.871348708Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000b234e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-nl2nr6" in resourcegroup="rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e",
      "name": "af93dcf2-82e0-4298-856d-f4846917b52e",
      "status": "Failed",
      "startTime": "2026-09-28T23:46:23.744577199Z",
      "endTime": "2026-09-29T00:05:30.871348708Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001664240>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001664240>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred

time=2026-09-28T22:45:44.302Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T22:53:20.758Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T22:45:44.312984171Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T22:45:46.293945727Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T22:45:46.372370612Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T22:53:20.755330702Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"93D44669EC2FD152\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j3781120-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT2M24.7661361S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755440055Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"D5BB7C72C0AE0DC5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j3781120-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT2M26.0225935S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755461175Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"3C41555BEADB3DE5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j3781120-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:45:49.7869316Z\",\"duration\":\"PT6M25.5435412S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755483916Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E73090DFF813B1FF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j3781120-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT1M41.3605417S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755497506Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DB7E7572BE861E73\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j3781120-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT1M37.1165773S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T22:53:20.755594868Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A1922F5B4A5E438F\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1\",\"resourceName\":\"ci01-j3781120-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:45:49.7869316Z\",\"duration\":\"PT5M2.9744241S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755630539Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109713389587325\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m36.442652668s\"}\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0012aa3f0>: 
    GET https://rp.j2517632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df74e584-612a-4664-98e5-95894a223dd0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df74e584-612a-4664-98e5-95894a223dd0",
      "name": "df74e584-612a-4664-98e5-95894a223dd0",
      "status": "Failed",
      "startTime": "2026-09-28T21:13:31.817925786Z",
      "endTime": "2026-09-28T21:42:34.60101187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0012aa3f0>: 
    GET https://rp.j2517632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df74e584-612a-4664-98e5-95894a223dd0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/df74e584-612a-4664-98e5-95894a223dd0",
      "name": "df74e584-612a-4664-98e5-95894a223dd0",
      "status": "Failed",
      "startTime": "2026-09-28T21:13:31.817925786Z",
      "endTime": "2026-09-28T21:42:34.60101187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T21:47:52Z
  Ended:       2026-09-28T22:02:50Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T21:44:52Z
  Ended:       2026-09-28T22:03:53Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T21:44:52Z
  Ended:       2026-09-28T22:00:53Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T20:22:39Z
  Ended:       2026-09-28T20:28:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9312256-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9312256-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc001512f00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-5f68mkqs6vtp" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-5f68mkqs6vtp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc001512f00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-5f68mkqs6vtp" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-5f68mkqs6vtp, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f1a240>: 
    resource group "rg-candidate-4-23-8lfxrmnvvv2w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-8hk4r9" in resourcegroup="rg-candidate-4-23-8lfxrmnvvv2w" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4",
      "name": "7972cdda-000f-4549-95ba-7b26044f70b4",
      "status": "Failed",
      "startTime": "2026-09-28T19:49:57.362020562Z",
      "endTime": "2026-09-28T20:14:00.292519621Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46r6c80n9jpgk27jtqr1mg2ert1grv still exists (deletion dispatched at 2026-09-28T19:49:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f1a240>: 
    resource group "rg-candidate-4-23-8lfxrmnvvv2w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-8hk4r9" in resourcegroup="rg-candidate-4-23-8lfxrmnvvv2w" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4",
      "name": "7972cdda-000f-4549-95ba-7b26044f70b4",
      "status": "Failed",
      "startTime": "2026-09-28T19:49:57.362020562Z",
      "endTime": "2026-09-28T20:14:00.292519621Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46r6c80n9jpgk27jtqr1mg2ert1grv still exists (deletion dispatched at 2026-09-28T19:49:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-b6k7qp"
Unexpected error:
    <*fmt.wrapError | 0xc000722f40>: 
    failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-b6k7qp"
Unexpected error:
    <*fmt.wrapError | 0xc000722f40>: 
    failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc00120a3a8>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zgxl8s4cd9df" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6724cdd6-177b-4593-9911-87900b80ceca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6724cdd6-177b-4593-9911-87900b80ceca",
      "name": "6724cdd6-177b-4593-9911-87900b80ceca",
      "status": "Failed",
      "startTime": "2026-09-28T19:42:59.042442346Z",
      "endTime": "2026-09-28T20:02:05.640354291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2k9vl-krm2l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zgxl8s4cd9df--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2k9vl-krm2l to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2k9vl-krm2l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zgxl8s4cd9df--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2k9vl-krm2l to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc00120a3a8>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zgxl8s4cd9df" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6724cdd6-177b-4593-9911-87900b80ceca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6724cdd6-177b-4593-9911-87900b80ceca",
      "name": "6724cdd6-177b-4593-9911-87900b80ceca",
      "status": "Failed",
      "startTime": "2026-09-28T19:42:59.042442346Z",
      "endTime": "2026-09-28T20:02:05.640354291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2k9vl-krm2l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zgxl8s4cd9df--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2k9vl-krm2l to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2k9vl-krm2l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zgxl8s4cd9df--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2k9vl-krm2l to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8fvg54" with version 4.22.15 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ae0680>: 
    failed to create HCP cluster cluster-zstream-4-22-8fvg54: failed waiting for cluster="cluster-zstream-4-22-8fvg54" in resourcegroup="rg-zstream-upgrade-4-22-8zfwm76xgnlj" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0",
      "name": "847235cf-8889-474c-8fc4-418de69191f0",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:18.057920805Z",
      "endTime": "2026-09-28T19:39:25.918712924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8fvg54" with version 4.22.15 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ae0680>: 
    failed to create HCP cluster cluster-zstream-4-22-8fvg54: failed waiting for cluster="cluster-zstream-4-22-8fvg54" in resourcegroup="rg-zstream-upgrade-4-22-8zfwm76xgnlj" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0",
      "name": "847235cf-8889-474c-8fc4-418de69191f0",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:18.057920805Z",
      "endTime": "2026-09-28T19:39:25.918712924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-27p526" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d20a60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-27p526: failed waiting for cluster="cp-ystream-upgrade-4-22-27p526" in resourcegroup="rg-cp-ystream-upgrade-4-22-7drm72sqb8zr" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "name": "3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:49.293974095Z",
      "endTime": "2026-09-28T19:39:55.897408712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-27p526" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d20a60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-27p526: failed waiting for cluster="cp-ystream-upgrade-4-22-27p526" in resourcegroup="rg-cp-ystream-upgrade-4-22-7drm72sqb8zr" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "name": "3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:49.293974095Z",
      "endTime": "2026-09-28T19:39:55.897408712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c241a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-c8jw5nmfkndx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808",
      "name": "4538241d-2021-40b7-8228-fb4052b1f808",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:37.948094059Z",
      "endTime": "2026-09-28T19:39:45.568048903Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c241a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-c8jw5nmfkndx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808",
      "name": "4538241d-2021-40b7-8228-fb4052b1f808",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:37.948094059Z",
      "endTime": "2026-09-28T19:39:45.568048903Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc001400360>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-sl29bx8fv8nz" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-sl29bx8fv8nz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc001400360>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-sl29bx8fv8nz" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-sl29bx8fv8nz, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000daa858>: 
    resource group "rg-hs-pre--zxrbtldmj8bt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-lz8thl" in resourcegroup="rg-hs-pre--zxrbtldmj8bt" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9657555-53b8-49b9-a717-afcd39c7e1e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9657555-53b8-49b9-a717-afcd39c7e1e0",
      "name": "a9657555-53b8-49b9-a717-afcd39c7e1e0",
      "status": "Failed",
      "startTime": "2026-09-28T19:30:58.262633809Z",
      "endTime": "2026-09-28T19:55:05.757251457Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46r69lkt992hqb9p61rfqbkob5804i still exists (deletion dispatched at 2026-09-28T19:30:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000daa858>: 
    resource group "rg-hs-pre--zxrbtldmj8bt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-lz8thl" in resourcegroup="rg-hs-pre--zxrbtldmj8bt" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9657555-53b8-49b9-a717-afcd39c7e1e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9657555-53b8-49b9-a717-afcd39c7e1e0",
      "name": "a9657555-53b8-49b9-a717-afcd39c7e1e0",
      "status": "Failed",
      "startTime": "2026-09-28T19:30:58.262633809Z",
      "endTime": "2026-09-28T19:55:05.757251457Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46r69lkt992hqb9p61rfqbkob5804i still exists (deletion dispatched at 2026-09-28T19:30:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6mht66l8hhvp/cluster-candidate-4-22-hxvn25 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00070f360>: 
    failed to create HCP cluster cluster-candidate-4-22-hxvn25: failed waiting for cluster="cluster-candidate-4-22-hxvn25" in resourcegroup="rg-candidate-4-22-6mht66l8hhvp" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "name": "889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:32.906905274Z",
      "endTime": "2026-09-28T19:39:35.528660951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6mht66l8hhvp/cluster-candidate-4-22-hxvn25 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00070f360>: 
    failed to create HCP cluster cluster-candidate-4-22-hxvn25: failed waiting for cluster="cluster-candidate-4-22-hxvn25" in resourcegroup="rg-candidate-4-22-6mht66l8hhvp" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "name": "889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:32.906905274Z",
      "endTime": "2026-09-28T19:39:35.528660951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002154570>: 
    resource group "complex-cilium-kv-jndv4cm2shzf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jndv4cm2shzf" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7",
      "name": "e40876ab-da07-40cc-bca8-2d36902295a7",
      "status": "Failed",
      "startTime": "2026-09-28T19:31:44.581294657Z",
      "endTime": "2026-09-28T19:55:45.796890355Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rtrm9n8ddf2ubgd68c1s64e1ioap still exists (deletion dispatched at 2026-09-28T19:31:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002154570>: 
    resource group "complex-cilium-kv-jndv4cm2shzf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jndv4cm2shzf" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7",
      "name": "e40876ab-da07-40cc-bca8-2d36902295a7",
      "status": "Failed",
      "startTime": "2026-09-28T19:31:44.581294657Z",
      "endTime": "2026-09-28T19:55:45.796890355Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rtrm9n8ddf2ubgd68c1s64e1ioap still exists (deletion dispatched at 2026-09-28T19:31:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0013e4180>: 
    resource group "private-keyvault-bzhkhkm6vpw7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-bzhkhkm6vpw7" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1",
      "name": "486f914e-d8e9-4103-988d-8630bbe515a1",
      "status": "Failed",
      "startTime": "2026-09-28T19:29:30.639536976Z",
      "endTime": "2026-09-28T19:53:35.789612068Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rt8qnc76ptkosa2u3fv39hps3u1f still exists (deletion dispatched at 2026-09-28T19:29:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0013e4180>: 
    resource group "private-keyvault-bzhkhkm6vpw7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-bzhkhkm6vpw7" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1",
      "name": "486f914e-d8e9-4103-988d-8630bbe515a1",
      "status": "Failed",
      "startTime": "2026-09-28T19:29:30.639536976Z",
      "endTime": "2026-09-28T19:53:35.789612068Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rt8qnc76ptkosa2u3fv39hps3u1f still exists (deletion dispatched at 2026-09-28T19:29:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc00060c440>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-br7vgttbd9zx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "name": "15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:44.986989982Z",
      "endTime": "2026-09-28T19:39:45.910063992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 24m39s ago; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc00060c440>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-br7vgttbd9zx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "name": "15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:44.986989982Z",
      "endTime": "2026-09-28T19:39:45.910063992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 24m39s ago; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4rl6n98blqls/cluster-candidate-5-0-9zqn4p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f22920>: 
    failed to create HCP cluster cluster-candidate-5-0-9zqn4p: failed waiting for cluster="cluster-candidate-5-0-9zqn4p" in resourcegroup="rg-candidate-5-0-4rl6n98blqls" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927",
      "name": "b919a89b-ddbd-409e-ae0b-825760357927",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:07.120894243Z",
      "endTime": "2026-09-28T19:39:16.873419842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4rl6n98blqls/cluster-candidate-5-0-9zqn4p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f22920>: 
    failed to create HCP cluster cluster-candidate-5-0-9zqn4p: failed waiting for cluster="cluster-candidate-5-0-9zqn4p" in resourcegroup="rg-candidate-5-0-4rl6n98blqls" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927",
      "name": "b919a89b-ddbd-409e-ae0b-825760357927",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:07.120894243Z",
      "endTime": "2026-09-28T19:39:16.873419842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010432a0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-66ncz7vqwm2z" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86",
      "name": "c35d648d-395d-429f-9f2d-24eed4314d86",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:11.81451702Z",
      "endTime": "2026-09-28T19:39:16.076223121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, community-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010432a0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-66ncz7vqwm2z" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86",
      "name": "c35d648d-395d-429f-9f2d-24eed4314d86",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:11.81451702Z",
      "endTime": "2026-09-28T19:39:16.076223121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, community-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bf61a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lhz84lf5nw65" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119",
      "name": "045ddd50-d43d-42a0-bcda-90888c378119",
      "status": "Failed",
      "startTime": "2026-09-28T19:11:03.945176993Z",
      "endTime": "2026-09-28T19:40:05.721943753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m29s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bf61a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lhz84lf5nw65" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119",
      "name": "045ddd50-d43d-42a0-bcda-90888c378119",
      "status": "Failed",
      "startTime": "2026-09-28T19:11:03.945176993Z",
      "endTime": "2026-09-28T19:40:05.721943753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m29s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 79 time(s) (63 unknown, 16 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:22:59Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="router-5b97c8c479-td767", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/router-5b97c8c479-td767 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T19:23:58Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rbkaemrkauofk92ubess3a0bmc8a-v20261001", pod="router-688fdf9d74-7b7qb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rbkaemrkauofk92ubess3a0bmc8a-v20261001/router-688fdf9d74-7b7qb has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T19:25:04Z
  Ended:       2026-09-28T19:47:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster", pod="router-7856497d5-p9s72", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster/router-7856497d5-p9s72 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T19:25:04Z
  Ended:       2026-09-28T19:47:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster", pod="router-7856497d5-ptl86", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster/router-7856497d5-ptl86 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="ovnkube-control-plane-5fd96b4979-gbmgt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/ovnkube-control-plane-5fd96b4979-gbmgt has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="ingress-operator-7d88f7b5dc-drxfd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/ingress-operator-7d88f7b5dc-drxfd has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-file-csi-driver-controller-854b478df8-v98lf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-file-csi-driver-controller-854b478df8-v98lf has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-disk-csi-driver-controller-5785d6645d-6xw4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-disk-csi-driver-controller-5785d6645d-6xw4f has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-disk-csi-driver-controller-5c84fd46c4-mpzts", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-disk-csi-driver-controller-5c84fd46c4-mpzts has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-file-csi-driver-controller-559d66d9f8-v468g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-file-csi-driver-controller-559d66d9f8-v468g has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-file-csi-driver-controller-9688c9fcd-vkkwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-file-csi-driver-controller-9688c9fcd-vkkwc has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-disk-csi-driver-controller-df8985bcf-xr5pq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-disk-csi-driver-controller-df8985bcf-xr5pq has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-file-csi-driver-controller-6f4b9c57cd-gpkcq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-file-csi-driver-controller-6f4b9c57cd-gpkcq has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-28T19:29:02Z
  Ended:       2026-09-28T19:35:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6qhsdqg59jb18mtn75mdepd994t-p1e0l2c6g7e7x9p", pod="azure-disk-csi-driver-controller-d5b4b46ff-swrjk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6qhsdqg59jb18mtn75mdepd994t-p1e0l2c6g7e7x9p/azure-disk-csi-driver-controller-d5b4b46ff-swrjk has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-28T19:29:02Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="ignition-server-6cf4c4cdbb-dcl2n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/ignition-server-6cf4c4cdbb-dcl2n has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="cluster-storage-operator-665d6949b6-6fw7x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/cluster-storage-operator-665d6949b6-6fw7x has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:37:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="ingress-operator-fb65c85-29jmc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/ingress-operator-fb65c85-29jmc has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="dns-operator-bfc958595-f2lvq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/dns-operator-bfc958595-f2lvq has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="kube-controller-manager-847bbfc79-gdtt6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/kube-controller-manager-847bbfc79-gdtt6 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="cluster-network-operator-6995f687db-6cfj4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/cluster-network-operator-6995f687db-6cfj4 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-28T19:33:01Z
  Ended:       2026-09-28T19:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rb6d2v8ktanhkbba2l3aora65d7u-t1b3k0j9k6s4g3e", pod="kube-apiserver-5f69cc7c69-66mrs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rb6d2v8ktanhkbba2l3aora65d7u-t1b3k0j9k6s4g3e/kube-apiserver-5f69cc7c69-66mrs has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-network-operator-79b745b7-2lm4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-network-operator-79b745b7-2lm4f has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="kube-apiserver-c7844b576-98nm5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/kube-apiserver-c7844b576-98nm5 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-version-operator-767f75f6bd-czn9v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-version-operator-767f75f6bd-czn9v has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="network-node-identity-688597d98d-5nwfj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/network-node-identity-688597d98d-5nwfj has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="kube-apiserver-674ffb8fcf-w2vnq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/kube-apiserver-674ffb8fcf-w2vnq has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="csi-snapshot-controller-78d9d968f6-lktl7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/csi-snapshot-controller-78d9d968f6-lktl7 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="kube-apiserver-7f4977d86c-ggtv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/kube-apiserver-7f4977d86c-ggtv4 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="community-operators-catalog-77957f8659-vrvjj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/community-operators-catalog-77957f8659-vrvjj has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="multus-admission-controller-6659c5d88f-hgsf6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/multus-admission-controller-6659c5d88f-hgsf6 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="router-7964b968d8-s9n7t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/router-7964b968d8-s9n7t has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="ignition-server-6cf4c4cdbb-xqn6v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/ignition-server-6cf4c4cdbb-xqn6v has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="openshift-apiserver-79bff85c55-rmr6j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/openshift-apiserver-79bff85c55-rmr6j has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="openshift-apiserver-f4f5cc6bf-j27d5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/openshift-apiserver-f4f5cc6bf-j27d5 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r619d7598ajep6f8mtkq1cmmun0r-g6f2d0d9e8y9l8w", pod="openshift-apiserver-855bf867d4-5dj69", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r619d7598ajep6f8mtkq1cmmun0r-g6f2d0d9e8y9l8w/openshift-apiserver-855bf867d4-5dj69 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="openshift-apiserver-548d95f77-j9rt4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/openshift-apiserver-548d95f77-j9rt4 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="kube-apiserver-8488cbccb8-9cntq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/kube-apiserver-8488cbccb8-9cntq has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="openshift-apiserver-59cf74f469-2pkbk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/openshift-apiserver-59cf74f469-2pkbk has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-image-registry-operator-5f9967bf67-m75d8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-image-registry-operator-5f9967bf67-m75d8 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="router-6966cb469f-jdx2g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/router-6966cb469f-jdx2g has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="kube-apiserver-6474ccdcd7-j5pkn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/kube-apiserver-6474ccdcd7-j5pkn has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="kube-apiserver-7877d486f4-dwd29", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/kube-apiserver-7877d486f4-dwd29 has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="network-node-identity-79f47f8db7-c6lsb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/network-node-identity-79f47f8db7-c6lsb has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="network-node-identity-6bf46b86d-r6lxm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/network-node-identity-6bf46b86d-r6lxm has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:48:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="router-d5fd8c684-npjp8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/router-d5fd8c684-npjp8 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="openshift-apiserver-7fccc5c996-2fqjh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/openshift-apiserver-7fccc5c996-2fqjh has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="certified-operators-catalog-56fcb687d5-x55km", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/certified-operators-catalog-56fcb687d5-x55km has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="openshift-apiserver-58689dc575-d9psz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/openshift-apiserver-58689dc575-d9psz has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="openshift-apiserver-fb96d7cd5-n8vx8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/openshift-apiserver-fb96d7cd5-n8vx8 has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="kube-apiserver-58866c8f4c-rssbm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/kube-apiserver-58866c8f4c-rssbm has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="openshift-apiserver-57597ccd79-kdsdf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/openshift-apiserver-57597ccd79-kdsdf has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="network-node-identity-6b45dfb8b5-gpsw6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/network-node-identity-6b45dfb8b5-gpsw6 has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="router-5b97c8c479-87d9j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/router-5b97c8c479-87d9j has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="network-node-identity-cf9fb8b88-4lqsp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/network-node-identity-cf9fb8b88-4lqsp has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="kube-apiserver-6776ddb8b4-jtcfw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/kube-apiserver-6776ddb8b4-jtcfw has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-09-28T19:36:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-file-csi-driver-controller-68d9cdbd6c-bjqhs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-file-csi-driver-controller-68d9cdbd6c-bjqhs has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-disk-csi-driver-controller-688579f7b8-gcwdq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-disk-csi-driver-controller-688579f7b8-gcwdq has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-disk-csi-driver-controller-778588d6c6-grn9w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-disk-csi-driver-controller-778588d6c6-grn9w has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="kube-controller-manager-5488d69fb6-p9ltk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/kube-controller-manager-5488d69fb6-p9ltk has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="kube-controller-manager-9864948dd-qzgnt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/kube-controller-manager-9864948dd-qzgnt has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="ovnkube-control-plane-58c8f9b6c5-45j6m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/ovnkube-control-plane-58c8f9b6c5-45j6m has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="network-node-identity-789b89cf4b-2v5sk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/network-node-identity-789b89cf4b-2v5sk has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-09-28T19:38:02Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re3hoaml91vd90m68oiaekbg44as-e7l1h2p9d0o6f4j", pod="openshift-apiserver-5f8cc55fd4-bm995", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re3hoaml91vd90m68oiaekbg44as-e7l1h2p9d0o6f4j/openshift-apiserver-5f8cc55fd4-bm995 has been in a non-ready state for longer than 5 minutes.

alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:37:02Z
  Ended:       2026-09-28T19:45:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:18:56Z
  Ended:       2026-09-28T19:51:55Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0189952-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0189952-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:45:04Z
  Ended:       2026-09-28T19:51:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="klusterlet-2t46re8cgt25tnc3e3r05ha4ola9p2m6", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t46re8cgt25tnc3e3r05ha4ola9p2m6/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002737518>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002737518>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullnsjbs8pwqtmk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00101eb60>: 
    VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e8a1ddc1-f3e1-4c99-ae14-bbe272e0b702; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7d311f24-e917-4750-a06e-94b359669923
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullnsjbs8pwqtmk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00101eb60>: 
    VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e8a1ddc1-f3e1-4c99-ae14-bbe272e0b702; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7d311f24-e917-4750-a06e-94b359669923
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T18:59:41Z
  Ended:       2026-09-28T19:05:43Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3028352-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3028352-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T18:17:15Z
  Ended:       2026-09-28T18:51:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6263424-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6263424-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T18:06:05Z
  Ended:       2026-09-28T18:40:06Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3070592-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3070592-svc).

time=2026-09-28T17:38:03.261Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T17:45:40.947Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T17:38:03.27081091Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T17:38:05.230722026Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T17:38:05.332101871Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T17:45:40.944777574Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"FDC79503A5389868\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j3078272-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT1M36.8422604S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"4\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 2, 1), 'Standard_E8ds_v6' (zones 1, 2), 'Standard_E8-4ds_v6' (zones 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T17:45:40.944878586Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DC248B23802810EC\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j3078272-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT1M22.2984424S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944892486Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F64F2B13FFA5C10A\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j3078272-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT2M10.1227436S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944904226Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"061A995FDE246942\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j3078272-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:38:09.8797046Z\",\"duration\":\"PT6M25.5728844S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944914146Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"4E79D1B051092E6E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j3078272-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT1M21.7193107S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944924847Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E245F41D17E48B12\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2\",\"resourceName\":\"ci01-j3078272-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:38:09.8797046Z\",\"duration\":\"PT5M2.6044727S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944945037Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109897999722695\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 2, 1), 'Standard_E8ds_v6' (zones 1, 2), 'Standard_E8-4ds_v6' (zones 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"4\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m37.674139897s\"}\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:183]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0xc000fb0020>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet kube-system/global-pull-secret-syncer has no desired pods scheduled
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:183]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0xc000fb0020>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet kube-system/global-pull-secret-syncer has no desired pods scheduled
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:755]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:755]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z"
Unexpected error:
    <*fmt.wrapError | 0xc000f712c0>: 
    failed to create NodePool np-wsvb2z: failed waiting for nodepool="np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z" in resourcegroup="rg-candidate-4-20-slrjsf69bdm7" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7",
      "name": "119c8117-1a73-44d6-8ce8-38b6990193b7",
      "status": "Failed",
      "startTime": "2026-09-28T17:43:51.079865069Z",
      "endTime": "2026-09-28T18:02:59.523800413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z"
Unexpected error:
    <*fmt.wrapError | 0xc000f712c0>: 
    failed to create NodePool np-wsvb2z: failed waiting for nodepool="np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z" in resourcegroup="rg-candidate-4-20-slrjsf69bdm7" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7",
      "name": "119c8117-1a73-44d6-8ce8-38b6990193b7",
      "status": "Failed",
      "startTime": "2026-09-28T17:43:51.079865069Z",
      "endTime": "2026-09-28T18:02:59.523800413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:476]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:476]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bb6980>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-v4wmmc9xsjpk" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bb6980>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-v4wmmc9xsjpk" to finish creating: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000ce94a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j9cxlp" in resourcegroup="rg-np-noedge-j9cxlp-9zvfnm962trj" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "name": "3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "status": "Failed",
      "startTime": "2026-09-28T17:41:00.477362299Z",
      "endTime": "2026-09-28T18:00:09.600349173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000ce94a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j9cxlp" in resourcegroup="rg-np-noedge-j9cxlp-9zvfnm962trj" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "name": "3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "status": "Failed",
      "startTime": "2026-09-28T17:41:00.477362299Z",
      "endTime": "2026-09-28T18:00:09.600349173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:615]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:615]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

time=2026-09-28T16:48:46.474Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T16:56:22.318Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T16:48:46.48313589Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T16:48:48.126456101Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T16:48:48.218251838Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T16:56:22.316196451Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"466F265DD247AC33\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j7409280-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:48:51.5604197Z\",\"duration\":\"PT7M11.5958475S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316251615Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"C2707792104BB37B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j7409280-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M38.4094883S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316265115Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"ED6706B9C7B9A085\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j7409280-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M54.5772258S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316275576Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9BFCDEDEF69AC484\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j7409280-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M20.8551291S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"3\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T16:56:22.316323959Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A08358469F613804\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j7409280-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M37.4251788S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.31633432Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F3D7EC585ED8A035\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2\",\"resourceName\":\"ci01-j7409280-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:48:51.5604197Z\",\"duration\":\"PT5M18.0784864S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316354711Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109927570685394\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"3\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m35.833223151s\"}\n exit status 1"

time=2026-09-28T16:41:53.539Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T16:50:00.699Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T16:41:53.548987458Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T16:41:55.657571892Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T16:41:55.746008508Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T16:50:00.696830163Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"6AAA4A6207C70729\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3611264-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j3611264-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:48:02.4739444Z\",\"duration\":\"PT1M21.6334503S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:50:00.696877604Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"68B6F701A0C34E31\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3611264-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j3611264-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:48:02.4739444Z\",\"duration\":\"PT1M53.575039S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:50:00.696896904Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"3B3E9A86D87CA201\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3611264-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j3611264-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T16:48:02.4739444Z\",\"duration\":\"PT34.2323713S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-64061740-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-64061740-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T16:50:00.696947105Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9C0D580618E74A6E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3611264-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j3611264-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:48:02.4739444Z\",\"duration\":\"PT1M21.1193253S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:50:00.696958195Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2D48FCCF3470DD4C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3611264-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j3611264-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:41:59.7167587Z\",\"duration\":\"PT7M55.4608511S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:50:00.696967625Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"BA32557B18EB08A6\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3611264-mgmt-1\",\"resourceName\":\"ci01-j3611264-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:41:59.7167587Z\",\"duration\":\"PT6M2.706804S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:50:00.696983806Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3611264-mgmt-1\",\"clusterName\":\"ci01-j3611264-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109931695561206\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3611264-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-64061740-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3611264-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-64061740-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"8m7.148003158s\"}\n exit status 1"

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T17:15:52Z
  Ended:       2026-09-28T17:50:53Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1544576-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1544576-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bltph8" for cluster "cluster-candidate-4-22-bltph8"
Unexpected error:
    <*fmt.wrapError | 0xc000f02a80>: 
    failed to create NodePool np-bltph8: failed waiting for nodepool="np-bltph8" for cluster "cluster-candidate-4-22-bltph8" in resourcegroup="rg-candidate-4-22-v77qxtrndwz7" to finish creating: GET https://rp.j4095744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/101f36de-888a-478f-8ea5-a713440e800f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/101f36de-888a-478f-8ea5-a713440e800f",
      "name": "101f36de-888a-478f-8ea5-a713440e800f",
      "status": "Failed",
      "startTime": "2026-09-28T17:28:01.310212925Z",
      "endTime": "2026-09-28T17:47:02.14138071Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-v77qxtrndwz7-candidate-bltph8--managed/providers/Microsoft.Compute/virtualMachines/c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-v77qxtrndwz7-candidate-bltph8--managed/providers/Microsoft.Compute/virtualMachines/c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bltph8" for cluster "cluster-candidate-4-22-bltph8"
Unexpected error:
    <*fmt.wrapError | 0xc000f02a80>: 
    failed to create NodePool np-bltph8: failed waiting for nodepool="np-bltph8" for cluster "cluster-candidate-4-22-bltph8" in resourcegroup="rg-candidate-4-22-v77qxtrndwz7" to finish creating: GET https://rp.j4095744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/101f36de-888a-478f-8ea5-a713440e800f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/101f36de-888a-478f-8ea5-a713440e800f",
      "name": "101f36de-888a-478f-8ea5-a713440e800f",
      "status": "Failed",
      "startTime": "2026-09-28T17:28:01.310212925Z",
      "endTime": "2026-09-28T17:47:02.14138071Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-v77qxtrndwz7-candidate-bltph8--managed/providers/Microsoft.Compute/virtualMachines/c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-v77qxtrndwz7-candidate-bltph8--managed/providers/Microsoft.Compute/virtualMachines/c7x6l7q7x4e1a8a-np-bltph8-pxvb4-5j9tr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:286]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc00080dc20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;
      - pod pull-secret-test container pull-test: ContainerCreating ()
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:286]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc00080dc20>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: 1 pod(s), no matching container with ImageID yet;
      - pod pull-secret-test container pull-test: ContainerCreating ()
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T17:11:53Z
  Ended:       2026-09-28T17:44:56Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2432512-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2432512-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:52:47Z
  Ended:       2026-09-28T17:00:49Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6875648-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6875648-svc).

time=2026-09-28T15:57:59.321Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T16:10:11.146Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T15:57:59.330300309Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T15:58:01.390991087Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T15:58:01.696498958Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T16:10:11.14280632Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"0311F1794B8012FE\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7129600-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j7129600-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:58:08.4615784Z\",\"duration\":\"PT11M40.7953789S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:10:11.142886205Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"5EA7D48F767F6548\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7129600-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j7129600-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T16:07:54.7264908Z\",\"duration\":\"PT1M38.5981419S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"1\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32015437-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32015437-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T16:10:11.14296792Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"6E7B2AA04B9A4637\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7129600-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j7129600-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:07:54.7264908Z\",\"duration\":\"PT1M22.2679526S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:10:11.142985571Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"AF3F03D52745E13A\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7129600-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j7129600-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:07:54.7264908Z\",\"duration\":\"PT1M53.5116434S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:10:11.143004953Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9ABBF5673F832862\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7129600-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j7129600-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:07:54.7264908Z\",\"duration\":\"PT1M21.0849543S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:10:11.143022244Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"43ED8335F40DFA22\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7129600-mgmt-1\",\"resourceName\":\"ci01-j7129600-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:58:08.4615784Z\",\"duration\":\"PT9M46.192924S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:10:11.143051885Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7129600-mgmt-1\",\"clusterName\":\"ci01-j7129600-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109958033794530\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7129600-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32015437-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7129600-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32015437-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"1\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"12m11.812755266s\"}\n exit status 1"

time=2026-09-28T15:59:28.426Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T16:06:34.696Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T15:59:28.435042187Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T15:59:30.367106783Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T15:59:30.626174897Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T16:06:34.694179167Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"4749BC6DF119391E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5424512-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j5424512-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:04:34.5902349Z\",\"duration\":\"PT1M23.0919369S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:06:34.694242988Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"067F3DF862DEE7AA\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5424512-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j5424512-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:59:33.8878249Z\",\"duration\":\"PT6M54.5799485S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:06:34.694260079Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"5F03570902E1ED24\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5424512-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j5424512-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T16:04:34.5902349Z\",\"duration\":\"PT34.8973447S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-93097460-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-93097460-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T16:06:34.69431613Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A4AE1C448D11E1F8\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5424512-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j5424512-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:04:34.5902349Z\",\"duration\":\"PT1M53.053524S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:06:34.69432878Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"98C15F1801605C4C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5424512-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j5424512-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:04:34.5902349Z\",\"duration\":\"PT1M21.7391537S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:06:34.69434204Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"BA6BD3DE4D4B26BF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5424512-mgmt-2\",\"resourceName\":\"ci01-j5424512-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:59:33.8878249Z\",\"duration\":\"PT5M0.6366021S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:06:34.694362061Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5424512-mgmt-2\",\"clusterName\":\"ci01-j5424512-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109957146694709\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5424512-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-93097460-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5424512-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-93097460-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m6.259324774s\"}\n exit status 1"

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:59:48Z
  Ended:       2026-09-28T17:05:47Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2598784-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2598784-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:58:54Z
  Ended:       2026-09-28T17:04:53Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6979328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6979328-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0003381e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-rvmnrkkl285p" to finish creating: GET https://rp.j9178240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c73ad27-aa36-40fa-b776-41e8f89f246a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c73ad27-aa36-40fa-b776-41e8f89f246a",
      "name": "9c73ad27-aa36-40fa-b776-41e8f89f246a",
      "status": "Failed",
      "startTime": "2026-09-28T16:28:51.042640714Z",
      "endTime": "2026-09-28T16:47:56.610289238Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-z6d2v-wmv2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-rvmnrkkl285p--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-z6d2v-wmv2z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-z6d2v-wmv2z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-rvmnrkkl285p--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-z6d2v-wmv2z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0003381e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-rvmnrkkl285p" to finish creating: GET https://rp.j9178240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c73ad27-aa36-40fa-b776-41e8f89f246a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c73ad27-aa36-40fa-b776-41e8f89f246a",
      "name": "9c73ad27-aa36-40fa-b776-41e8f89f246a",
      "status": "Failed",
      "startTime": "2026-09-28T16:28:51.042640714Z",
      "endTime": "2026-09-28T16:47:56.610289238Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-z6d2v-wmv2z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-rvmnrkkl285p--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-z6d2v-wmv2z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-z6d2v-wmv2z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-rvmnrkkl285p--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-z6d2v-wmv2z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:56:14Z
  Ended:       2026-09-28T17:04:13Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9178240-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9178240-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-sxjtd8" for cluster "cluster-hs-pre-sxjtd8"
Unexpected error:
    <*fmt.wrapError | 0xc001083460>: 
    failed to create NodePool np-sxjtd8: failed waiting for nodepool="np-sxjtd8" for cluster "cluster-hs-pre-sxjtd8" in resourcegroup="rg-hs-pre--b9q689nq2wdz" to finish creating: GET https://rp.j8087296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8a6222c-2fba-4e87-a303-c9f2805ff8b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8a6222c-2fba-4e87-a303-c9f2805ff8b9",
      "name": "e8a6222c-2fba-4e87-a303-c9f2805ff8b9",
      "status": "Failed",
      "startTime": "2026-09-28T16:22:48.273113179Z",
      "endTime": "2026-09-28T16:41:51.753830234Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--b9q689nq2wdz-sxjtd8--managed/providers/Microsoft.Compute/virtualMachines/y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--b9q689nq2wdz-sxjtd8--managed/providers/Microsoft.Compute/virtualMachines/y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-sxjtd8" for cluster "cluster-hs-pre-sxjtd8"
Unexpected error:
    <*fmt.wrapError | 0xc001083460>: 
    failed to create NodePool np-sxjtd8: failed waiting for nodepool="np-sxjtd8" for cluster "cluster-hs-pre-sxjtd8" in resourcegroup="rg-hs-pre--b9q689nq2wdz" to finish creating: GET https://rp.j8087296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8a6222c-2fba-4e87-a303-c9f2805ff8b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8a6222c-2fba-4e87-a303-c9f2805ff8b9",
      "name": "e8a6222c-2fba-4e87-a303-c9f2805ff8b9",
      "status": "Failed",
      "startTime": "2026-09-28T16:22:48.273113179Z",
      "endTime": "2026-09-28T16:41:51.753830234Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--b9q689nq2wdz-sxjtd8--managed/providers/Microsoft.Compute/virtualMachines/y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--b9q689nq2wdz-sxjtd8--managed/providers/Microsoft.Compute/virtualMachines/y0n4l3m3f1d2x9m-np-sxjtd8-978kp-mhmdc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:57:44Z
  Ended:       2026-09-28T17:03:45Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7085696-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7085696-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:53:13Z
  Ended:       2026-09-28T16:59:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1117440-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1117440-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:38:15Z
  Ended:       2026-09-28T16:46:11Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0662784-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0662784-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:20:58Z
  Ended:       2026-09-28T16:27:58Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0662784-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 39 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qwlf6", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qwlf6 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-w28wh", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-w28wh has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-zxx8r", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-zxx8r has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-g76qj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-g76qj has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-d9qzz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-d9qzz has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-fpkv2", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-fpkv2 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-cwzx8", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-cwzx8 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qm6c8", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qm6c8 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-4996z", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-4996z has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-675f5", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-675f5 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-gd9k9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-gd9k9 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-z58gp", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-z58gp has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-4j2hv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-4j2hv has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-8n4fj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-8n4fj has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-8q8gs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-8q8gs has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-ddxpz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-ddxpz has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-9kqwf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-9kqwf has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-tpk28", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-tpk28 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-wpwxf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-wpwxf has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-c9pkv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-c9pkv has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-r2z8h", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-r2z8h has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-v6lst", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-v6lst has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-6nc22", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-6nc22 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-52zbx", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-52zbx has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-j8nbh", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-j8nbh has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-f9lhx", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-f9lhx has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dp559", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dp559 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-j4pwc", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-j4pwc has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-p9w7c", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-p9w7c has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-hqdp9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-hqdp9 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dgb65", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dgb65 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-rvk8h", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-rvk8h has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-mpfhk", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-mpfhk has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qm2ts", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qm2ts has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-27cpw", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-27cpw has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-mxsvz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-mxsvz has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-hgtjs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-hgtjs has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-c8pzf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-c8pzf has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dvzkd", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dvzkd has been in a non-ready state for longer than 5 minutes.

time=2026-09-28T15:01:37.564Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-09-28T15:01:39.047Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-09-28T15:01:41.809Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=07143feb5a4416edfd80f78e9bd52b0ace9e0b4259fb3b361ffbc72b09db2a4e portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8532096-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F07143feb5a4416edfd80f78e9bd52b0ace9e0b4259fb3b361ffbc72b09db2a4e
time=2026-09-28T15:06:13.242Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8532096-svc/providers/Microsoft.Resources/deployments/07143feb5a4416edfd80f78e9bd52b0ace9e0b4259fb3b361ffbc72b09db2a4e/operationStatuses/08584109991856512016\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8532096-svc/providers/Microsoft.Resources/deployments/cluster-fgxbml5ybyqlw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8532096-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j8532096-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n                \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8532096-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-system-29994265-vmss'. Please try alternative sizes 'Standard_D4as_v7' (zones 3, 1, 2), 'Standard_D4ads_v7' (zones 1, 2, 3), 'Standard_E4as_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8532096-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-system-29994265-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\r\\n                    \\\"target\\\": \\\"2\\\",\\r\\n                    \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-28T14:56:44.121Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T15:04:20.182Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T14:56:44.133408984Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T14:56:45.999747591Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T14:56:46.087122797Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T15:04:20.179772906Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"6904C144EA6E5071\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2572160-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j2572160-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:02:20.3817346Z\",\"duration\":\"PT1M58.3746655S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T15:04:20.179836397Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A4BA1CEE66306436\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2572160-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j2572160-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:02:20.3817346Z\",\"duration\":\"PT1M37.5722982S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T15:04:20.179850277Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"410417A2FC6329C6\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2572160-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j2572160-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T14:56:49.1631026Z\",\"duration\":\"PT7M8.6010377S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-17691843-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-17691843-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T15:04:20.179898728Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"70FD9837C10AD01B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2572160-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j2572160-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:02:20.3817346Z\",\"duration\":\"PT1M21.3213377S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T15:04:20.179909679Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"8940E8E2A38F1650\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2572160-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j2572160-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T15:02:20.3817346Z\",\"duration\":\"PT1M52.0121162S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T15:04:20.179919029Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"01D23ED14CA92F00\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2572160-mgmt-1\",\"resourceName\":\"ci01-j2572160-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T14:56:49.1631026Z\",\"duration\":\"PT5M31.1376288S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T15:04:20.179947539Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2572160-mgmt-1\",\"clusterName\":\"ci01-j2572160-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109994792416073\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j2572160-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-17691843-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2572160-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-17691843-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m36.046545346s\"}\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullhctmh8kwn4gw.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00011dc00>: 
    VerifyImagePulled(acrpullhctmh8kwn4gw.azurecr.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullhctmh8kwn4gw.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00011dc00>: 
    VerifyImagePulled(acrpullhctmh8kwn4gw.azurecr.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T15:42:33Z
  Ended:       2026-09-28T15:48:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0815872-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0815872-svc).

time=2026-09-28T14:47:30.090Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T14:54:36.446Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T14:47:30.098885868Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T14:47:31.998596058Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T14:47:32.096190187Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T14:54:36.443742575Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"7422A9B9C0E00015\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0415360-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j0415360-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T14:52:06.8114069Z\",\"duration\":\"PT1M22.9729442S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T14:54:36.443804706Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A288211471CD0A68\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0415360-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j0415360-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T14:52:06.8114069Z\",\"duration\":\"PT2M25.5163011S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"1\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-40607818-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-40607818-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T14:54:36.443859458Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"67AAD81D54389E82\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0415360-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j0415360-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T14:47:35.7788484Z\",\"duration\":\"PT6M26.5731388S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T14:54:36.443870708Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E019F45794232502\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0415360-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j0415360-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T14:52:06.8114069Z\",\"duration\":\"PT1M52.7194683S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T14:54:36.443881869Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"BF186E56B03A4C4B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0415360-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j0415360-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T14:52:06.8114069Z\",\"duration\":\"PT1M36.8059515S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T14:54:36.443891779Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"532DC9DFA113CABD\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0415360-mgmt-2\",\"resourceName\":\"ci01-j0415360-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T14:47:35.7788484Z\",\"duration\":\"PT4M30.9311267S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T14:54:36.44391733Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0415360-mgmt-2\",\"clusterName\":\"ci01-j0415360-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584110000332060255\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0415360-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-40607818-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0415360-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-40607818-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"1\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m6.345035722s\"}\n exit status 1"

time=2026-09-28T14:59:06.467Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-09-28T14:59:07.052Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:bd6cc099-ebdb-45c9-b00c-0802e69adff6 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-ci01-j9638656] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j9638656.hcpsvc.osadev.cloud] image:map[digest:sha256:e46e58946095455f1186c85136cc576df37e30746f4cbf6630f40ee15e5c51e6 registry:arohcpsvcdev.azurecr.io repository:ci-op-w348pxjm/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-w348pxjm/pipeline:pull workloadIdentityClientId:78d6364f-6f7b-4fee-9060-b833e4581820 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:85e8e804-1ab2-4c98-b7ae-b44472a84e8a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-09-28T14:59:07.052Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-09-28T14:59:07.261Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate
time=2026-09-28T14:59:07.573Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false
time=2026-09-28T14:59:07.573Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-09-28T14:59:07.688Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-09-28T14:59:15.529Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-29.aks-istio-system.svc:443/validate?timeout=10s\": EOF"
time=2026-09-28T14:59:15.529Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-29.aks-istio-system.svc:443/validate?timeout=10s\": EOF"

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T15:37:38Z
  Ended:       2026-09-28T15:43:41Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1684736-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1684736-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T14:24:12Z
  Ended:       2026-09-28T14:56:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3283328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3283328-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ca2e60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-z4h7jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-z4h7jf/operationStatuses/08584110018702456355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name/providers/Microsoft.Authorization/roleAssignments/ce4c794c-50e9-565e-99cc-b7dd7e08c5a9' at line '1' and column '29054'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 29054,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          },\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/e5697265-1ef6-54a3-8e48-b96178ae0611' at line '1' and column '30683'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 30683,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ca2e60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-z4h7jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-z4h7jf/operationStatuses/08584110018702456355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name/providers/Microsoft.Authorization/roleAssignments/ce4c794c-50e9-565e-99cc-b7dd7e08c5a9' at line '1' and column '29054'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 29054,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          },\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/e5697265-1ef6-54a3-8e48-b96178ae0611' at line '1' and column '30683'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 30683,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T14:42:44Z
  Ended:       2026-09-28T14:48:45Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6224640-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6224640-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-bq4xzd" for cluster "cluster-hs-pre-bq4xzd"
Unexpected error:
    <*fmt.wrapError | 0xc000a77280>: 
    failed to create NodePool np-bq4xzd: failed waiting for nodepool="np-bq4xzd" for cluster "cluster-hs-pre-bq4xzd" in resourcegroup="rg-hs-pre--t5k7dv9kqkn6" to finish creating: GET https://rp.j4365952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7ac9caa-24fd-4f24-be91-2c74f17d3115
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7ac9caa-24fd-4f24-be91-2c74f17d3115",
      "name": "c7ac9caa-24fd-4f24-be91-2c74f17d3115",
      "status": "Failed",
      "startTime": "2026-09-28T13:36:17.28936251Z",
      "endTime": "2026-09-28T13:55:25.887167153Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--t5k7dv9kqkn6-bq4xzd--managed/providers/Microsoft.Compute/virtualMachines/e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--t5k7dv9kqkn6-bq4xzd--managed/providers/Microsoft.Compute/virtualMachines/e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-bq4xzd" for cluster "cluster-hs-pre-bq4xzd"
Unexpected error:
    <*fmt.wrapError | 0xc000a77280>: 
    failed to create NodePool np-bq4xzd: failed waiting for nodepool="np-bq4xzd" for cluster "cluster-hs-pre-bq4xzd" in resourcegroup="rg-hs-pre--t5k7dv9kqkn6" to finish creating: GET https://rp.j4365952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7ac9caa-24fd-4f24-be91-2c74f17d3115
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7ac9caa-24fd-4f24-be91-2c74f17d3115",
      "name": "c7ac9caa-24fd-4f24-be91-2c74f17d3115",
      "status": "Failed",
      "startTime": "2026-09-28T13:36:17.28936251Z",
      "endTime": "2026-09-28T13:55:25.887167153Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--t5k7dv9kqkn6-bq4xzd--managed/providers/Microsoft.Compute/virtualMachines/e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--t5k7dv9kqkn6-bq4xzd--managed/providers/Microsoft.Compute/virtualMachines/e6l4u8u5m8k2h1y-np-bq4xzd-k5mvt-twpcj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:33:38Z
  Ended:       2026-09-28T14:05:42Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4365952-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4365952-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001373000>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k8wr4fpfcrsg" to finish creating: GET https://rp.j6155008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf7814ee-225c-4e86-a66b-262e98fa9add
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf7814ee-225c-4e86-a66b-262e98fa9add",
      "name": "cf7814ee-225c-4e86-a66b-262e98fa9add",
      "status": "Failed",
      "startTime": "2026-09-28T13:36:07.829911324Z",
      "endTime": "2026-09-28T13:55:12.891520361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-k8wr4fpfcrsg--managed/providers/Microsoft.Compute/virtualMachines/u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-k8wr4fpfcrsg--managed/providers/Microsoft.Compute/virtualMachines/u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001373000>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k8wr4fpfcrsg" to finish creating: GET https://rp.j6155008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf7814ee-225c-4e86-a66b-262e98fa9add
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf7814ee-225c-4e86-a66b-262e98fa9add",
      "name": "cf7814ee-225c-4e86-a66b-262e98fa9add",
      "status": "Failed",
      "startTime": "2026-09-28T13:36:07.829911324Z",
      "endTime": "2026-09-28T13:55:12.891520361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-k8wr4fpfcrsg--managed/providers/Microsoft.Compute/virtualMachines/u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-k8wr4fpfcrsg--managed/providers/Microsoft.Compute/virtualMachines/u9v0q3j6a3y7e1k-np-1-cg9ct-bj9h7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:58:43Z
  Ended:       2026-09-28T14:07:44Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6155008-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6155008-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000693320>: 
    DNS for route host agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000693320>: 
    DNS for route host agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T14:36:28Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j1676288-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.70:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1676288/providers/microsoft.monitor/accounts/services-j1676288", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6c86cd5988-85nd2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j1676288-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T14:36:28Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j1676288-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.70:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1676288/providers/microsoft.monitor/accounts/services-j1676288", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6c86cd5988-85nd2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j1676288-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

time=2026-09-28T12:07:41.041Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T12:15:18.546Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T12:07:41.053161868Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T12:07:43.329595506Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T12:07:43.531590773Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T12:15:18.541524871Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"AD2AC9DE0BA65092\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5699200-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j5699200-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T12:13:09.5885526Z\",\"duration\":\"PT1M22.2941476S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T12:15:18.541698705Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"795D33A61BFD5FC4\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5699200-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j5699200-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T12:13:09.5885526Z\",\"duration\":\"PT1M23.0194131S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T12:15:18.541724696Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B79CD1DE1D75E03E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5699200-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j5699200-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T12:13:09.5885526Z\",\"duration\":\"PT1M55.2502144S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T12:15:18.541752616Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1C6C75893BADF622\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5699200-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j5699200-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T12:13:09.5885526Z\",\"duration\":\"PT1M53.2791501S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T12:15:18.541784267Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"097072C37326C4E8\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5699200-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j5699200-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T12:07:47.6806894Z\",\"duration\":\"PT6M43.9724254S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"2\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16955988-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16955988-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T12:15:18.541891629Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"6469ADD3757B8F13\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5699200-mgmt-2\",\"resourceName\":\"ci01-j5699200-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T12:07:47.6806894Z\",\"duration\":\"PT5M21.6490011S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T12:15:18.541955731Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5699200-mgmt-2\",\"clusterName\":\"ci01-j5699200-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584110096217749999\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j5699200-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16955988-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5699200-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16955988-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"2\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m37.488796782s\"}\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-c89c6r" with version 5.0.0-rc.3 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b10480>: 
    failed to create HCP cluster cluster-zstream-5-0-c89c6r: failed waiting for cluster="cluster-zstream-5-0-c89c6r" in resourcegroup="rg-zstream-upgrade-5-0-6kc9p4vcvjqq" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "name": "37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:45.206355596Z",
      "endTime": "2026-09-28T13:22:51.172490782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 26m23s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-c89c6r" with version 5.0.0-rc.3 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b10480>: 
    failed to create HCP cluster cluster-zstream-5-0-c89c6r: failed waiting for cluster="cluster-zstream-5-0-c89c6r" in resourcegroup="rg-zstream-upgrade-5-0-6kc9p4vcvjqq" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "name": "37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:45.206355596Z",
      "endTime": "2026-09-28T13:22:51.172490782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 26m23s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0004bd2a8>: 
    resource group "rg-candidate-4-21-phpjd5zrm7d6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-5p8bwm" in resourcegroup="rg-candidate-4-21-phpjd5zrm7d6" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8e63bd1c-cb86-4f54-9bc3-fd86e73fbb80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8e63bd1c-cb86-4f54-9bc3-fd86e73fbb80",
      "name": "8e63bd1c-cb86-4f54-9bc3-fd86e73fbb80",
      "status": "Failed",
      "startTime": "2026-09-28T13:20:19.409419895Z",
      "endTime": "2026-09-28T13:44:21.013354483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41am2lgq0mrglm8sdpilfbg6dbc207 still exists (deletion dispatched at 2026-09-28T13:20:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0004bd2a8>: 
    resource group "rg-candidate-4-21-phpjd5zrm7d6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-5p8bwm" in resourcegroup="rg-candidate-4-21-phpjd5zrm7d6" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8e63bd1c-cb86-4f54-9bc3-fd86e73fbb80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8e63bd1c-cb86-4f54-9bc3-fd86e73fbb80",
      "name": "8e63bd1c-cb86-4f54-9bc3-fd86e73fbb80",
      "status": "Failed",
      "startTime": "2026-09-28T13:20:19.409419895Z",
      "endTime": "2026-09-28T13:44:21.013354483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41am2lgq0mrglm8sdpilfbg6dbc207 still exists (deletion dispatched at 2026-09-28T13:20:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0xc000ea49c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-q4hdh7csw5qs" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508",
      "name": "daab790a-4984-45ab-a2a3-a3f37cd79508",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:48.885600114Z",
      "endTime": "2026-09-28T13:22:50.894287651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m10s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0xc000ea49c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-q4hdh7csw5qs" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508",
      "name": "daab790a-4984-45ab-a2a3-a3f37cd79508",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:48.885600114Z",
      "endTime": "2026-09-28T13:22:50.894287651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m10s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d60180>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-txbk9bmdh8wg" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "name": "7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:22.508354202Z",
      "endTime": "2026-09-28T13:22:30.229773982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 26m20s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d60180>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-txbk9bmdh8wg" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "name": "7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:22.508354202Z",
      "endTime": "2026-09-28T13:22:30.229773982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 26m20s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012643d8>: 
    resource group "rg-candidate-4-20-5mk9bsj2ctvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-rp6nvl" in resourcegroup="rg-candidate-4-20-5mk9bsj2ctvs" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b24fc68-0b47-4d43-852a-f452e6006f54
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b24fc68-0b47-4d43-852a-f452e6006f54",
      "name": "7b24fc68-0b47-4d43-852a-f452e6006f54",
      "status": "Failed",
      "startTime": "2026-09-28T13:14:15.967505225Z",
      "endTime": "2026-09-28T13:38:21.093544627Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41ah00kund6k88mndce49mop4aje17 still exists (deletion dispatched at 2026-09-28T13:14:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012643d8>: 
    resource group "rg-candidate-4-20-5mk9bsj2ctvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-rp6nvl" in resourcegroup="rg-candidate-4-20-5mk9bsj2ctvs" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b24fc68-0b47-4d43-852a-f452e6006f54
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b24fc68-0b47-4d43-852a-f452e6006f54",
      "name": "7b24fc68-0b47-4d43-852a-f452e6006f54",
      "status": "Failed",
      "startTime": "2026-09-28T13:14:15.967505225Z",
      "endTime": "2026-09-28T13:38:21.093544627Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41ah00kund6k88mndce49mop4aje17 still exists (deletion dispatched at 2026-09-28T13:14:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e3b0e0>: 
    resource group "rg-candidate-4-22-pb89drlbvrsv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-prcc26" in resourcegroup="rg-candidate-4-22-pb89drlbvrsv" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/144216a6-d87d-43b0-9650-80be413ca8c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/144216a6-d87d-43b0-9650-80be413ca8c9",
      "name": "144216a6-d87d-43b0-9650-80be413ca8c9",
      "status": "Failed",
      "startTime": "2026-09-28T13:19:57.5093726Z",
      "endTime": "2026-09-28T13:44:00.980312186Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41agprs3sucbbjdath1ukr197cicdg still exists (deletion dispatched at 2026-09-28T13:20:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e3b0e0>: 
    resource group "rg-candidate-4-22-pb89drlbvrsv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-prcc26" in resourcegroup="rg-candidate-4-22-pb89drlbvrsv" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/144216a6-d87d-43b0-9650-80be413ca8c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/144216a6-d87d-43b0-9650-80be413ca8c9",
      "name": "144216a6-d87d-43b0-9650-80be413ca8c9",
      "status": "Failed",
      "startTime": "2026-09-28T13:19:57.5093726Z",
      "endTime": "2026-09-28T13:44:00.980312186Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41agprs3sucbbjdath1ukr197cicdg still exists (deletion dispatched at 2026-09-28T13:20:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-26r6nt" with version 4.20.39 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106e580>: 
    failed to create HCP cluster cluster-zstream-4-20-26r6nt: failed waiting for cluster="cluster-zstream-4-20-26r6nt" in resourcegroup="rg-zstream-upgrade-4-20-6bpd6fpsn6s4" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "name": "4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:43.832400833Z",
      "endTime": "2026-09-28T13:22:51.48967289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-apiserver, openshift-route-controller-manager, catalog-operator, community-operators-catalog, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-storage-operator, openshift-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, ingress-operator, cluster-policy-controller, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-26r6nt" with version 4.20.39 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106e580>: 
    failed to create HCP cluster cluster-zstream-4-20-26r6nt: failed waiting for cluster="cluster-zstream-4-20-26r6nt" in resourcegroup="rg-zstream-upgrade-4-20-6bpd6fpsn6s4" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "name": "4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:43.832400833Z",
      "endTime": "2026-09-28T13:22:51.48967289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-apiserver, openshift-route-controller-manager, catalog-operator, community-operators-catalog, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-storage-operator, openshift-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, ingress-operator, cluster-policy-controller, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--gd5x92vx9cnh/cluster-hs-pre-r2s4dx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fd2480>: 
    failed to create HCP cluster cluster-hs-pre-r2s4dx: failed waiting for cluster="cluster-hs-pre-r2s4dx" in resourcegroup="rg-hs-pre--gd5x92vx9cnh" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "name": "471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:54.032352928Z",
      "endTime": "2026-09-28T13:23:00.490629254Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--gd5x92vx9cnh/cluster-hs-pre-r2s4dx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fd2480>: 
    failed to create HCP cluster cluster-hs-pre-r2s4dx: failed waiting for cluster="cluster-hs-pre-r2s4dx" in resourcegroup="rg-hs-pre--gd5x92vx9cnh" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "name": "471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:54.032352928Z",
      "endTime": "2026-09-28T13:23:00.490629254Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc002725230>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w8pbb4pwwmm7" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-w8pbb4pwwmm7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc002725230>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w8pbb4pwwmm7" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-w8pbb4pwwmm7, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e5c108>: 
    resource group "v20261001preview-qdw6lx4gfk7z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-qdw6lx4gfk7z" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ee12444a-4944-4551-91b1-efd928e4d331
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ee12444a-4944-4551-91b1-efd928e4d331",
      "name": "ee12444a-4944-4551-91b1-efd928e4d331",
      "status": "Failed",
      "startTime": "2026-09-28T13:20:48.465989676Z",
      "endTime": "2026-09-28T13:44:50.987334733Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41ah3qate5m8su42c0558vhm7cr3k1 still exists (deletion dispatched at 2026-09-28T13:20:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e5c108>: 
    resource group "v20261001preview-qdw6lx4gfk7z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-qdw6lx4gfk7z" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ee12444a-4944-4551-91b1-efd928e4d331
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ee12444a-4944-4551-91b1-efd928e4d331",
      "name": "ee12444a-4944-4551-91b1-efd928e4d331",
      "status": "Failed",
      "startTime": "2026-09-28T13:20:48.465989676Z",
      "endTime": "2026-09-28T13:44:50.987334733Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41ah3qate5m8su42c0558vhm7cr3k1 still exists (deletion dispatched at 2026-09-28T13:20:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000dbafa0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-xh7nfd4qdxnv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a",
      "name": "10cf3d15-2823-4e15-9280-d4bdf793742a",
      "status": "Failed",
      "startTime": "2026-09-28T12:54:54.12824002Z",
      "endTime": "2026-09-28T13:24:00.287293188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000dbafa0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-xh7nfd4qdxnv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a",
      "name": "10cf3d15-2823-4e15-9280-d4bdf793742a",
      "status": "Failed",
      "startTime": "2026-09-28T12:54:54.12824002Z",
      "endTime": "2026-09-28T13:24:00.287293188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a6bae0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5w2hj7t2fgmv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "name": "fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:32.768656716Z",
      "endTime": "2026-09-28T13:22:40.579367905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m26s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a6bae0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5w2hj7t2fgmv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "name": "fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:32.768656716Z",
      "endTime": "2026-09-28T13:22:40.579367905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m26s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000ef8198>: 
    resource group "feature-aggregation-xs2z7sms8dlg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-xs2z7sms8dlg" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/54660575-548c-4595-879d-9a906af13f0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/54660575-548c-4595-879d-9a906af13f0e",
      "name": "54660575-548c-4595-879d-9a906af13f0e",
      "status": "Failed",
      "startTime": "2026-09-28T13:24:12.67709827Z",
      "endTime": "2026-09-28T13:48:20.982209436Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41bq34gqtdianb4nt2b6ogejjvitge still exists (deletion dispatched at 2026-09-28T13:24:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000ef8198>: 
    resource group "feature-aggregation-xs2z7sms8dlg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-xs2z7sms8dlg" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/54660575-548c-4595-879d-9a906af13f0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/54660575-548c-4595-879d-9a906af13f0e",
      "name": "54660575-548c-4595-879d-9a906af13f0e",
      "status": "Failed",
      "startTime": "2026-09-28T13:24:12.67709827Z",
      "endTime": "2026-09-28T13:48:20.982209436Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41bq34gqtdianb4nt2b6ogejjvitge still exists (deletion dispatched at 2026-09-28T13:24:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a36160>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-znb5fvxdcbts" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8be6cab6-4516-4904-95c9-f95cbd718be0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8be6cab6-4516-4904-95c9-f95cbd718be0",
      "name": "8be6cab6-4516-4904-95c9-f95cbd718be0",
      "status": "Failed",
      "startTime": "2026-09-28T12:55:42.029686391Z",
      "endTime": "2026-09-28T13:24:50.589882071Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m31s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a36160>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-znb5fvxdcbts" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8be6cab6-4516-4904-95c9-f95cbd718be0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8be6cab6-4516-4904-95c9-f95cbd718be0",
      "name": "8be6cab6-4516-4904-95c9-f95cbd718be0",
      "status": "Failed",
      "startTime": "2026-09-28T12:55:42.029686391Z",
      "endTime": "2026-09-28T13:24:50.589882071Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m31s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001f043c0>: 
    resource group "rg-candidate-4-23-4pqspxzwdtwh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-kgnzqx" in resourcegroup="rg-candidate-4-23-4pqspxzwdtwh" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ff9462-d02e-4eaf-a164-a96ef15aebfd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ff9462-d02e-4eaf-a164-a96ef15aebfd",
      "name": "02ff9462-d02e-4eaf-a164-a96ef15aebfd",
      "status": "Failed",
      "startTime": "2026-09-28T13:18:50.813351207Z",
      "endTime": "2026-09-28T13:42:51.002385237Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41ao91k1fdh4d610bhaes144ec06jp still exists (deletion dispatched at 2026-09-28T13:18:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001f043c0>: 
    resource group "rg-candidate-4-23-4pqspxzwdtwh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-kgnzqx" in resourcegroup="rg-candidate-4-23-4pqspxzwdtwh" to finish deleting: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ff9462-d02e-4eaf-a164-a96ef15aebfd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ff9462-d02e-4eaf-a164-a96ef15aebfd",
      "name": "02ff9462-d02e-4eaf-a164-a96ef15aebfd",
      "status": "Failed",
      "startTime": "2026-09-28T13:18:50.813351207Z",
      "endTime": "2026-09-28T13:42:51.002385237Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t41ao91k1fdh4d610bhaes144ec06jp still exists (deletion dispatched at 2026-09-28T13:18:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:09:10Z
  Ended:       2026-09-28T13:39:13Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:06:06Z
  Ended:       2026-09-28T13:14:06Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:01:03Z
  Ended:       2026-09-28T13:36:01Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8288256-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8288256-svc).

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:18:05Z
  Ended:       2026-09-28T13:31:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T13:18:05Z
  Ended:       2026-09-28T13:29:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:21:03Z
  Ended:       2026-09-28T13:27:01Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="prometheus", node="aks-userswft3-30800232-vmss000000", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft3-30800232-vmss000000 is unreachable and some workloads may be rescheduled.

alert fired 54 time(s) (38 unknown, 16 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:06:01Z
  Ended:       2026-09-28T13:29:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41bfp4l0n502nn624hggcdkif3fakm-l7r5c9r0t4i6i9i", pod="router-5f8dd5cdc8-2x7hv", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41bfp4l0n502nn624hggcdkif3fakm-l7r5c9r0t4i6i9i/router-5f8dd5cdc8-2x7hv has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:13:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="cluster-network-operator-745b94c8d6-xgbrm", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/cluster-network-operator-745b94c8d6-xgbrm has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:29:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster", pod="router-7f5958d584-h7qql", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster/router-7f5958d584-h7qql has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="openshift-apiserver-77dbbb84d4-g84qr", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/openshift-apiserver-77dbbb84d4-g84qr has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41aorgo7k4e5t9d3gl5386u8qrr1ok-b1g2t3h0x4w9w8x", pod="router-869d798bbd-2c6kt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41aorgo7k4e5t9d3gl5386u8qrr1ok-b1g2t3h0x4w9w8x/router-869d798bbd-2c6kt has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:37:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster", pod="router-7f5958d584-xsp4j", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster/router-7f5958d584-xsp4j has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-28T13:08:02Z
  Ended:       2026-09-28T13:35:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41bju2fkvns9a7dk4sqfvummalvm0f-t0e8p3i4e9p7l9u", pod="router-6449799f6-vwm42", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41bju2fkvns9a7dk4sqfvummalvm0f-t0e8p3i4e9p7l9u/router-6449799f6-vwm42 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-28T13:08:02Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="ignition-server-proxy-67ffb7567d-ndltn", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/ignition-server-proxy-67ffb7567d-ndltn has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-28T13:08:03Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41an05susenrnl0dfs40rcas06ch4c-f7z5i4t9g6o0n5p", pod="ignition-server-proxy-8699545598-fvwm5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41an05susenrnl0dfs40rcas06ch4c-f7z5i4t9g6o0n5p/ignition-server-proxy-8699545598-fvwm5 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-28T13:08:03Z
  Ended:       2026-09-28T13:26:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-controller-manager-867d6bcbc8-7zqfz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-controller-manager-867d6bcbc8-7zqfz has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-28T13:16:02Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="openshift-apiserver-db5f58d87-n2wk7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/openshift-apiserver-db5f58d87-n2wk7 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="kube-apiserver-85d5846f96-gpf28", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/kube-apiserver-85d5846f96-gpf28 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", pod="openshift-apiserver-85677f65c4-wrvhg", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics/openshift-apiserver-85677f65c4-wrvhg has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="network-node-identity-7549b9758b-4wb6s", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/network-node-identity-7549b9758b-4wb6s has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="openshift-apiserver-77dbbb84d4-29r9g", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/openshift-apiserver-77dbbb84d4-29r9g has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-apiserver-5c98b4447b-wvxv8", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-apiserver-5c98b4447b-wvxv8 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="kube-apiserver-745ff9b4db-srgc7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/kube-apiserver-745ff9b4db-srgc7 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="openshift-apiserver-84fcf4bc4d-mcmsx", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/openshift-apiserver-84fcf4bc4d-mcmsx has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="openshift-apiserver-b6b777476-w4j8h", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/openshift-apiserver-b6b777476-w4j8h has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="network-node-identity-57484c8884-trf2c", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/network-node-identity-57484c8884-trf2c has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="openshift-apiserver-5c68654d4b-dbjjt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/openshift-apiserver-5c68654d4b-dbjjt has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="openshift-apiserver-68c54f7686-xjxd2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/openshift-apiserver-68c54f7686-xjxd2 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="kube-apiserver-86c59c88cb-qrxm8", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/kube-apiserver-86c59c88cb-qrxm8 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-controller-manager-867d6bcbc8-bpsb5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-controller-manager-867d6bcbc8-bpsb5 has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="network-node-identity-65ff54759c-92pcp", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/network-node-identity-65ff54759c-92pcp has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="network-node-identity-594c4dccf4-lm2n6", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/network-node-identity-594c4dccf4-lm2n6 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", pod="kube-apiserver-b8d4f754f-dqwb6", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics/kube-apiserver-b8d4f754f-dqwb6 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="kube-apiserver-957c77ffc-rg8wb", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/kube-apiserver-957c77ffc-rg8wb has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="kube-apiserver-5c9ff4f96c-n5vjw", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/kube-apiserver-5c9ff4f96c-n5vjw has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="openshift-apiserver-554784946b-mjxzz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/openshift-apiserver-554784946b-mjxzz has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="kube-apiserver-65f96cf6cc-d9g85", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/kube-apiserver-65f96cf6cc-d9g85 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-28T13:19:08Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anli60mk7obvb9qscekiln1r7nlh-z0b5k9r4h8u4l9q", pod="openshift-apiserver-5f4c5b7f7c-l5gpc", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anli60mk7obvb9qscekiln1r7nlh-z0b5k9r4h8u4l9q/openshift-apiserver-5f4c5b7f7c-l5gpc has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-28T13:20:08Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="multus-admission-controller-78665bcc7c-5f2bh", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/multus-admission-controller-78665bcc7c-5f2bh has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="csi-snapshot-controller-operator-74759598d-8dhlz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/csi-snapshot-controller-operator-74759598d-8dhlz has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="cluster-storage-operator-55cc4d6675-r7fxd", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/cluster-storage-operator-55cc4d6675-r7fxd has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="cluster-network-operator-5dc56d94dc-rmrjd", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/cluster-network-operator-5dc56d94dc-rmrjd has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="dns-operator-6d45bd4dc-c5h9t", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/dns-operator-6d45bd4dc-c5h9t has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="ingress-operator-6c4557f7b9-x9crg", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/ingress-operator-6c4557f7b9-x9crg has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:06:05Z
  Ended:       2026-09-28T13:14:04Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:09:09Z
  Ended:       2026-09-28T13:23:06Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:57:33Z
  Ended:       2026-09-28T12:43:33Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:57:33Z
  Ended:       2026-09-28T12:43:33Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T12:00:53Z
  Ended:       2026-09-28T12:14:51Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0514944-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T12:06:51Z
  Ended:       2026-09-28T12:18:52Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0514944-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:54:31Z
  Ended:       2026-09-28T12:18:28Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:54:32Z
  Ended:       2026-09-28T12:18:31Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T12:03:41Z
  Ended:       2026-09-28T12:36:41Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1304448-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1304448-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T12:25:41Z
  Ended:       2026-09-28T12:31:44Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1304448-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

time=2026-09-28T11:34:50.835Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T11:47:58.887Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T11:34:50.844839709Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T11:34:53.110680532Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T11:34:53.249915799Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T11:47:58.883999384Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"68FDB0D02588B7A4\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7452672-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j7452672-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T11:45:45.8915868Z\",\"duration\":\"PT1M54.7878497S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T11:47:58.884100077Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2C437C987C8C8646\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7452672-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j7452672-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T11:45:45.8915868Z\",\"duration\":\"PT1M37.708543S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-30789729-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-30789729-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T11:47:58.8841827Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1D4A4ACD08AFCCDC\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7452672-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j7452672-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T11:34:57.5557466Z\",\"duration\":\"PT12M43.6290559S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T11:47:58.88420525Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"6F7320C9245928C1\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7452672-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j7452672-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T11:45:45.8915868Z\",\"duration\":\"PT1M21.9184137S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T11:47:58.884223601Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"873F83FBC798DF6B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7452672-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j7452672-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T11:45:45.8915868Z\",\"duration\":\"PT1M22.7299182S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T11:47:58.884239541Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"D548DA7D42E659A4\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7452672-mgmt-1\",\"resourceName\":\"ci01-j7452672-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T11:34:57.5557466Z\",\"duration\":\"PT10M47.9375428S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T11:47:58.884272442Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7452672-mgmt-1\",\"clusterName\":\"ci01-j7452672-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584110115920308752\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j7452672-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-30789729-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7452672-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-30789729-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"13m8.039438803s\"}\n exit status 1"

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T12:26:58Z
  Ended:       2026-09-28T12:37:54Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3789824-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3789824-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:21:38Z
  Ended:       2026-09-28T11:54:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7241856-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7241856-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:44:58Z
  Ended:       2026-09-28T12:14:55Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:24:54Z
  Ended:       2026-09-28T11:45:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:27:55Z
  Ended:       2026-09-28T12:10:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:41:58Z
  Ended:       2026-09-28T11:49:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:22:20Z
  Ended:       2026-09-28T12:10:21Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j0144000-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3vp4t5eip9aogjiq8jka0asjt0oqsm-private-kas", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0144000-mgmt-2
Namespace: ocm-arohcpci01-2t3vp4t5eip9aogjiq8jka0asjt0oqsm-private-kas

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:19:18Z
  Ended:       2026-09-28T11:45:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j0144000-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3vp4t5eip9aogjiq8jka0asjt0oqsm-private-kas", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0144000-mgmt-2
Namespace: ocm-arohcpci01-2t3vp4t5eip9aogjiq8jka0asjt0oqsm-private-kas

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-htjpjq" for cluster "cluster-hs-pre-htjpjq"
Unexpected error:
    <*fmt.wrapError | 0xc000a9e360>: 
    failed to create NodePool np-htjpjq: failed waiting for nodepool="np-htjpjq" for cluster "cluster-hs-pre-htjpjq" in resourcegroup="rg-hs-pre--7q459bkmgf27" to finish creating: GET https://rp.j0955136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1c41955-68c5-43d7-a640-907c30bcadb1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1c41955-68c5-43d7-a640-907c30bcadb1",
      "name": "d1c41955-68c5-43d7-a640-907c30bcadb1",
      "status": "Failed",
      "startTime": "2026-09-28T10:29:10.039353403Z",
      "endTime": "2026-09-28T10:48:12.965050693Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--7q459bkmgf27-htjpjq--managed/providers/Microsoft.Compute/virtualMachines/u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--7q459bkmgf27-htjpjq--managed/providers/Microsoft.Compute/virtualMachines/u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-htjpjq" for cluster "cluster-hs-pre-htjpjq"
Unexpected error:
    <*fmt.wrapError | 0xc000a9e360>: 
    failed to create NodePool np-htjpjq: failed waiting for nodepool="np-htjpjq" for cluster "cluster-hs-pre-htjpjq" in resourcegroup="rg-hs-pre--7q459bkmgf27" to finish creating: GET https://rp.j0955136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1c41955-68c5-43d7-a640-907c30bcadb1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d1c41955-68c5-43d7-a640-907c30bcadb1",
      "name": "d1c41955-68c5-43d7-a640-907c30bcadb1",
      "status": "Failed",
      "startTime": "2026-09-28T10:29:10.039353403Z",
      "endTime": "2026-09-28T10:48:12.965050693Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--7q459bkmgf27-htjpjq--managed/providers/Microsoft.Compute/virtualMachines/u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--7q459bkmgf27-htjpjq--managed/providers/Microsoft.Compute/virtualMachines/u6f9w7w4s9l9e8y-np-htjpjq-hqvsc-tsmn8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:53:59Z
  Ended:       2026-09-28T11:00:01Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0955136-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0955136-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:40:21Z
  Ended:       2026-09-28T11:23:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j3233536-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3v2u1i7qsc0v7mcq5u2f0jotsatk8u-d2u4p8t3s9k5z2b", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j3233536-mgmt-2
Namespace: ocm-arohcpci01-2t3v2u1i7qsc0v7mcq5u2f0jotsatk8u-d2u4p8t3s9k5z2b

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:32:27Z
  Ended:       2026-09-28T10:58:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j3233536-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3v2u1i7qsc0v7mcq5u2f0jotsatk8u-d2u4p8t3s9k5z2b", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j3233536-mgmt-2
Namespace: ocm-arohcpci01-2t3v2u1i7qsc0v7mcq5u2f0jotsatk8u-d2u4p8t3s9k5z2b

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:36:45Z
  Ended:       2026-09-28T10:54:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3233536-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T10:36:45Z
  Ended:       2026-09-28T10:57:52Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3233536-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:52:35Z
  Ended:       2026-09-28T11:00:37Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0068736-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0068736-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:33:36Z
  Ended:       2026-09-28T10:45:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7088128-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:53:56Z
  Ended:       2026-09-28T11:00:56Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7088128-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7088128-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00138a980>: 
    failed to create HCP cluster np-dg-minor-w2lmrh: failed waiting for cluster="np-dg-minor-w2lmrh" in resourcegroup="rg-np-dg-minor-w2lmrh-zzht7hmgmswz" to finish creating: GET https://rp.j2515584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a4adc4d-0ad3-406d-a9bc-9c1ae046c034
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a4adc4d-0ad3-406d-a9bc-9c1ae046c034",
      "name": "8a4adc4d-0ad3-406d-a9bc-9c1ae046c034",
      "status": "Failed",
      "startTime": "2026-09-28T10:35:16.490368834Z",
      "endTime": "2026-09-28T11:04:17.550861433Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 1m2s ago; version 4.22.16 is Partial (want Completed), started 7m32s ago; version 4.22.16 is Partial (want Completed), started 11m2s ago; version 4.22.16 is Partial (want Completed), started 18m25s ago; version 4.22.16 is Partial (want Completed), started 20m51s ago; version 4.22.16 is Partial (want Completed), started 23m2s ago; version 4.22.16 is Partial (want Completed), started 26m19s ago; version 4.22.16 is Partial (want Completed), started 27m32s ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00138a980>: 
    failed to create HCP cluster np-dg-minor-w2lmrh: failed waiting for cluster="np-dg-minor-w2lmrh" in resourcegroup="rg-np-dg-minor-w2lmrh-zzht7hmgmswz" to finish creating: GET https://rp.j2515584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a4adc4d-0ad3-406d-a9bc-9c1ae046c034
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a4adc4d-0ad3-406d-a9bc-9c1ae046c034",
      "name": "8a4adc4d-0ad3-406d-a9bc-9c1ae046c034",
      "status": "Failed",
      "startTime": "2026-09-28T10:35:16.490368834Z",
      "endTime": "2026-09-28T11:04:17.550861433Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 1m2s ago; version 4.22.16 is Partial (want Completed), started 7m32s ago; version 4.22.16 is Partial (want Completed), started 11m2s ago; version 4.22.16 is Partial (want Completed), started 18m25s ago; version 4.22.16 is Partial (want Completed), started 20m51s ago; version 4.22.16 is Partial (want Completed), started 23m2s ago; version 4.22.16 is Partial (want Completed), started 26m19s ago; version 4.22.16 is Partial (want Completed), started 27m32s ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:17:06Z
  Ended:       2026-09-28T10:51:06Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2515584-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2515584-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.008s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001488a20>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:46491->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.008s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001488a20>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:46491->172.30.0.10:53: read: connection refused
    ...

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:47:15Z
  Ended:       2026-09-28T10:31:14Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4135168-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3ucelqfhl68luucdn21goolqq53uj5-private-kas", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4135168-mgmt-1
Namespace: ocm-arohcpci01-2t3ucelqfhl68luucdn21goolqq53uj5-private-kas

Firing 2:
  State:       Resolved
  Started:     2026-09-28T09:49:15Z
  Ended:       2026-09-28T10:04:14Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4135168-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3uchfsr0akt7a7mrha2rd3io4p1b79-g6g2o0e4x6a3e2d", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4135168-mgmt-2
Namespace: ocm-arohcpci01-2t3uchfsr0akt7a7mrha2rd3io4p1b79-g6g2o0e4x6a3e2d

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:44:14Z
  Ended:       2026-09-28T10:06:14Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4135168-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3ucelqfhl68luucdn21goolqq53uj5-private-kas", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4135168-mgmt-1
Namespace: ocm-arohcpci01-2t3ucelqfhl68luucdn21goolqq53uj5-private-kas

Firing 2:
  State:       Resolved
  Started:     2026-09-28T09:46:17Z
  Ended:       2026-09-28T10:02:11Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4135168-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3uchfsr0akt7a7mrha2rd3io4p1b79-g6g2o0e4x6a3e2d", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4135168-mgmt-2
Namespace: ocm-arohcpci01-2t3uchfsr0akt7a7mrha2rd3io4p1b79-g6g2o0e4x6a3e2d

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

time=2026-09-28T09:01:02.051Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T09:08:09.651Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T09:01:02.061522961Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T09:01:04.147818707Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T09:01:04.440466024Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T09:08:09.64760036Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9614EBA97D5F9800\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4798720-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j4798720-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T09:05:41.381016Z\",\"duration\":\"PT1M37.2773647S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16665582-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16665582-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T09:08:09.647751843Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"66FF17855A3FEDFC\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4798720-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j4798720-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T09:05:41.381016Z\",\"duration\":\"PT2M25.6050717S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T09:08:09.647775174Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"160C685E1A59B997\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4798720-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j4798720-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T09:01:08.8659112Z\",\"duration\":\"PT5M55.8315068S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T09:08:09.647794924Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B1467C06696E9BAC\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4798720-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j4798720-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T09:05:41.381016Z\",\"duration\":\"PT1M22.0799243S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T09:08:09.647811774Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"50E19B7C817D66F4\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4798720-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j4798720-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T09:05:41.381016Z\",\"duration\":\"PT2M24.6050443S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T09:08:09.647834635Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B0D1165F6FF7B302\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4798720-mgmt-2\",\"resourceName\":\"ci01-j4798720-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T09:01:08.8659112Z\",\"duration\":\"PT4M32.2580835S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T09:08:09.647869416Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4798720-mgmt-2\",\"clusterName\":\"ci01-j4798720-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584110208208621976\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4798720-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16665582-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4798720-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-16665582-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m7.586350686s\"}\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00172e090>: 
    resource group "complex-cilium-kv-k9knbpmhbg7v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k9knbpmhbg7v" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "name": "4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:43.521207454Z",
      "endTime": "2026-09-28T09:54:45.223507829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u15umjtvbfn7or4kh25ftp6mt8ubm still exists (deletion dispatched at 2026-09-28T09:30:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00172e090>: 
    resource group "complex-cilium-kv-k9knbpmhbg7v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k9knbpmhbg7v" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "name": "4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:43.521207454Z",
      "endTime": "2026-09-28T09:54:45.223507829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u15umjtvbfn7or4kh25ftp6mt8ubm still exists (deletion dispatched at 2026-09-28T09:30:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c06120>: 
    resource group "rg-np-version-upgrade-8l6tr2-428422fmlt4m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-8l6tr2" in resourcegroup="rg-np-version-upgrade-8l6tr2-428422fmlt4m" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9bc1619-e6b7-4915-a5af-afeb34d9739c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9bc1619-e6b7-4915-a5af-afeb34d9739c",
      "name": "f9bc1619-e6b7-4915-a5af-afeb34d9739c",
      "status": "Failed",
      "startTime": "2026-09-28T09:47:50.839437919Z",
      "endTime": "2026-09-28T10:11:59.725052507Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0lq4iolgmmgdclkhq6jg6ede2jbm still exists (deletion dispatched at 2026-09-28T09:47:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c06120>: 
    resource group "rg-np-version-upgrade-8l6tr2-428422fmlt4m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-8l6tr2" in resourcegroup="rg-np-version-upgrade-8l6tr2-428422fmlt4m" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9bc1619-e6b7-4915-a5af-afeb34d9739c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9bc1619-e6b7-4915-a5af-afeb34d9739c",
      "name": "f9bc1619-e6b7-4915-a5af-afeb34d9739c",
      "status": "Failed",
      "startTime": "2026-09-28T09:47:50.839437919Z",
      "endTime": "2026-09-28T10:11:59.725052507Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0lq4iolgmmgdclkhq6jg6ede2jbm still exists (deletion dispatched at 2026-09-28T09:47:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001010288>: 
    resource group "clusternp128-cr6m72hbhthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cr6m72hbhthl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "name": "2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "status": "Failed",
      "startTime": "2026-09-28T09:26:14.153004619Z",
      "endTime": "2026-09-28T09:50:15.107483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ep16n5al4fc1ggfac6imb350l still exists (deletion dispatched at 2026-09-28T09:26:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001010288>: 
    resource group "clusternp128-cr6m72hbhthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cr6m72hbhthl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "name": "2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "status": "Failed",
      "startTime": "2026-09-28T09:26:14.153004619Z",
      "endTime": "2026-09-28T09:50:15.107483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ep16n5al4fc1ggfac6imb350l still exists (deletion dispatched at 2026-09-28T09:26:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a50120>: 
    resource group "nodepool-delete-dfzrjzvh57lj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dfzrjzvh57lj" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87e2a749-adb1-4e6e-9c80-163809fbf472
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87e2a749-adb1-4e6e-9c80-163809fbf472",
      "name": "87e2a749-adb1-4e6e-9c80-163809fbf472",
      "status": "Failed",
      "startTime": "2026-09-28T09:40:50.481180787Z",
      "endTime": "2026-09-28T10:04:59.753897901Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ioqrfjsn6acu7pjpg12lkcddspk still exists (deletion dispatched at 2026-09-28T09:40:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a50120>: 
    resource group "nodepool-delete-dfzrjzvh57lj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dfzrjzvh57lj" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87e2a749-adb1-4e6e-9c80-163809fbf472
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87e2a749-adb1-4e6e-9c80-163809fbf472",
      "name": "87e2a749-adb1-4e6e-9c80-163809fbf472",
      "status": "Failed",
      "startTime": "2026-09-28T09:40:50.481180787Z",
      "endTime": "2026-09-28T10:04:59.753897901Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ioqrfjsn6acu7pjpg12lkcddspk still exists (deletion dispatched at 2026-09-28T09:40:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f12198>: 
    resource group "feature-aggregation-qfgsqjtw55cd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-qfgsqjtw55cd" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3913a806-5810-4793-abd7-5ef178b7415c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3913a806-5810-4793-abd7-5ef178b7415c",
      "name": "3913a806-5810-4793-abd7-5ef178b7415c",
      "status": "Failed",
      "startTime": "2026-09-28T09:37:58.390921917Z",
      "endTime": "2026-09-28T10:02:05.045669677Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u1jrr7vnhpej35p0tpr9a3np9a5ic still exists (deletion dispatched at 2026-09-28T09:38:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f12198>: 
    resource group "feature-aggregation-qfgsqjtw55cd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-qfgsqjtw55cd" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3913a806-5810-4793-abd7-5ef178b7415c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3913a806-5810-4793-abd7-5ef178b7415c",
      "name": "3913a806-5810-4793-abd7-5ef178b7415c",
      "status": "Failed",
      "startTime": "2026-09-28T09:37:58.390921917Z",
      "endTime": "2026-09-28T10:02:05.045669677Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u1jrr7vnhpej35p0tpr9a3np9a5ic still exists (deletion dispatched at 2026-09-28T09:38:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001722090>: 
    resource group "rg-zstream-upgrade-4-20-r5d6pqgs2lcz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-xfrrpx" in resourcegroup="rg-zstream-upgrade-4-20-r5d6pqgs2lcz" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/142a98c7-5abe-44fb-90f2-c5de9bb7a00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/142a98c7-5abe-44fb-90f2-c5de9bb7a00e",
      "name": "142a98c7-5abe-44fb-90f2-c5de9bb7a00e",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:23.402647295Z",
      "endTime": "2026-09-28T10:00:25.321453256Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ffhfsef977jtdd70rv0ev42cvrg still exists (deletion dispatched at 2026-09-28T09:36:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001722090>: 
    resource group "rg-zstream-upgrade-4-20-r5d6pqgs2lcz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-xfrrpx" in resourcegroup="rg-zstream-upgrade-4-20-r5d6pqgs2lcz" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/142a98c7-5abe-44fb-90f2-c5de9bb7a00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/142a98c7-5abe-44fb-90f2-c5de9bb7a00e",
      "name": "142a98c7-5abe-44fb-90f2-c5de9bb7a00e",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:23.402647295Z",
      "endTime": "2026-09-28T10:00:25.321453256Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ffhfsef977jtdd70rv0ev42cvrg still exists (deletion dispatched at 2026-09-28T09:36:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001178498>: 
    resource group "rg-np-skip-minor-dw8b66-7pzf52lvmpf4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-dw8b66" in resourcegroup="rg-np-skip-minor-dw8b66-7pzf52lvmpf4" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55b08ad7-ac72-45f5-9b5d-12339524531b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55b08ad7-ac72-45f5-9b5d-12339524531b",
      "name": "55b08ad7-ac72-45f5-9b5d-12339524531b",
      "status": "Failed",
      "startTime": "2026-09-28T09:46:43.12833531Z",
      "endTime": "2026-09-28T10:10:49.874421326Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0lvk9h4cv1mk6uo8iiei2pbj0pmg still exists (deletion dispatched at 2026-09-28T09:46:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001178498>: 
    resource group "rg-np-skip-minor-dw8b66-7pzf52lvmpf4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-dw8b66" in resourcegroup="rg-np-skip-minor-dw8b66-7pzf52lvmpf4" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55b08ad7-ac72-45f5-9b5d-12339524531b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55b08ad7-ac72-45f5-9b5d-12339524531b",
      "name": "55b08ad7-ac72-45f5-9b5d-12339524531b",
      "status": "Failed",
      "startTime": "2026-09-28T09:46:43.12833531Z",
      "endTime": "2026-09-28T10:10:49.874421326Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0lvk9h4cv1mk6uo8iiei2pbj0pmg still exists (deletion dispatched at 2026-09-28T09:46:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000bc80c0>: 
    resource group "rg-cp-ystream-upgrade-4-23-wzvvqkwwvxlw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-23-2zhgtf" in resourcegroup="rg-cp-ystream-upgrade-4-23-wzvvqkwwvxlw" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d382c657-1215-4712-a751-4d1a4f16b05f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d382c657-1215-4712-a751-4d1a4f16b05f",
      "name": "d382c657-1215-4712-a751-4d1a4f16b05f",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:53.979369262Z",
      "endTime": "2026-09-28T10:00:55.003547453Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ic75rjvskjgforinth6afhbjsc5 still exists (deletion dispatched at 2026-09-28T09:36:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000bc80c0>: 
    resource group "rg-cp-ystream-upgrade-4-23-wzvvqkwwvxlw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-23-2zhgtf" in resourcegroup="rg-cp-ystream-upgrade-4-23-wzvvqkwwvxlw" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d382c657-1215-4712-a751-4d1a4f16b05f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d382c657-1215-4712-a751-4d1a4f16b05f",
      "name": "d382c657-1215-4712-a751-4d1a4f16b05f",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:53.979369262Z",
      "endTime": "2026-09-28T10:00:55.003547453Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ic75rjvskjgforinth6afhbjsc5 still exists (deletion dispatched at 2026-09-28T09:36:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000898240>: 
    resource group "oidc-wi-6h5hr74nbxbl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-6h5hr74nbxbl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de033d5e-0c9a-4b34-a70f-e0babef6ad67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de033d5e-0c9a-4b34-a70f-e0babef6ad67",
      "name": "de033d5e-0c9a-4b34-a70f-e0babef6ad67",
      "status": "Failed",
      "startTime": "2026-09-28T09:48:11.942517946Z",
      "endTime": "2026-09-28T10:12:19.666093496Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubc77sob0k75ihn39813gvminerda still exists (deletion dispatched at 2026-09-28T09:48:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000898240>: 
    resource group "oidc-wi-6h5hr74nbxbl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-6h5hr74nbxbl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de033d5e-0c9a-4b34-a70f-e0babef6ad67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de033d5e-0c9a-4b34-a70f-e0babef6ad67",
      "name": "de033d5e-0c9a-4b34-a70f-e0babef6ad67",
      "status": "Failed",
      "startTime": "2026-09-28T09:48:11.942517946Z",
      "endTime": "2026-09-28T10:12:19.666093496Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubc77sob0k75ihn39813gvminerda still exists (deletion dispatched at 2026-09-28T09:48:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f7e360>: 
    resource group "v20260901preview-2tskgvphk5pz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-2tskgvphk5pz" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b9b932-13e2-4612-b6b0-df57f3cc7eb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b9b932-13e2-4612-b6b0-df57f3cc7eb8",
      "name": "13b9b932-13e2-4612-b6b0-df57f3cc7eb8",
      "status": "Failed",
      "startTime": "2026-09-28T09:31:10.812279232Z",
      "endTime": "2026-09-28T09:55:14.912402863Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0mu3m9kv9273jhipsv94emihr3l8 still exists (deletion dispatched at 2026-09-28T09:31:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f7e360>: 
    resource group "v20260901preview-2tskgvphk5pz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-2tskgvphk5pz" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b9b932-13e2-4612-b6b0-df57f3cc7eb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b9b932-13e2-4612-b6b0-df57f3cc7eb8",
      "name": "13b9b932-13e2-4612-b6b0-df57f3cc7eb8",
      "status": "Failed",
      "startTime": "2026-09-28T09:31:10.812279232Z",
      "endTime": "2026-09-28T09:55:14.912402863Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0mu3m9kv9273jhipsv94emihr3l8 still exists (deletion dispatched at 2026-09-28T09:31:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00165a8b8>: 
    resource group "np-autoscaling-d78mqd29jkc5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-d78mqd29jkc5" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fd7cab-045b-483c-9285-d64f8be28bf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fd7cab-045b-483c-9285-d64f8be28bf8",
      "name": "17fd7cab-045b-483c-9285-d64f8be28bf8",
      "status": "Failed",
      "startTime": "2026-09-28T09:37:39.043957536Z",
      "endTime": "2026-09-28T10:01:44.896160867Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0mtuah2idr1sqhvfksccie7qk7bk still exists (deletion dispatched at 2026-09-28T09:37:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00165a8b8>: 
    resource group "np-autoscaling-d78mqd29jkc5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-d78mqd29jkc5" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fd7cab-045b-483c-9285-d64f8be28bf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fd7cab-045b-483c-9285-d64f8be28bf8",
      "name": "17fd7cab-045b-483c-9285-d64f8be28bf8",
      "status": "Failed",
      "startTime": "2026-09-28T09:37:39.043957536Z",
      "endTime": "2026-09-28T10:01:44.896160867Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0mtuah2idr1sqhvfksccie7qk7bk still exists (deletion dispatched at 2026-09-28T09:37:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fa2390>: 
    resource group "rg-np-labels-taints-xb88nprh6v8r": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-xb88nprh6v8r" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a0c6ea6-d19d-46df-ab9e-9088ec00bcf2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a0c6ea6-d19d-46df-ab9e-9088ec00bcf2",
      "name": "9a0c6ea6-d19d-46df-ab9e-9088ec00bcf2",
      "status": "Failed",
      "startTime": "2026-09-28T09:35:33.916123706Z",
      "endTime": "2026-09-28T09:59:35.59252554Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0lp2q6dsf21b8209r6h4nf0vedlp still exists (deletion dispatched at 2026-09-28T09:35:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fa2390>: 
    resource group "rg-np-labels-taints-xb88nprh6v8r": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-xb88nprh6v8r" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a0c6ea6-d19d-46df-ab9e-9088ec00bcf2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9a0c6ea6-d19d-46df-ab9e-9088ec00bcf2",
      "name": "9a0c6ea6-d19d-46df-ab9e-9088ec00bcf2",
      "status": "Failed",
      "startTime": "2026-09-28T09:35:33.916123706Z",
      "endTime": "2026-09-28T09:59:35.59252554Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0lp2q6dsf21b8209r6h4nf0vedlp still exists (deletion dispatched at 2026-09-28T09:35:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a32078>: 
    resource group "rg-cp-ystream-upgrade-4-22-4lhbjmsv9rt8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-47qvqf" in resourcegroup="rg-cp-ystream-upgrade-4-22-4lhbjmsv9rt8" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e635505a-5ce4-4b08-a35e-5c1ae209525b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e635505a-5ce4-4b08-a35e-5c1ae209525b",
      "name": "e635505a-5ce4-4b08-a35e-5c1ae209525b",
      "status": "Failed",
      "startTime": "2026-09-28T09:37:59.959232804Z",
      "endTime": "2026-09-28T10:02:05.043620822Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ma6360plq8ebh66osj03lihjgik still exists (deletion dispatched at 2026-09-28T09:38:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a32078>: 
    resource group "rg-cp-ystream-upgrade-4-22-4lhbjmsv9rt8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-47qvqf" in resourcegroup="rg-cp-ystream-upgrade-4-22-4lhbjmsv9rt8" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e635505a-5ce4-4b08-a35e-5c1ae209525b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e635505a-5ce4-4b08-a35e-5c1ae209525b",
      "name": "e635505a-5ce4-4b08-a35e-5c1ae209525b",
      "status": "Failed",
      "startTime": "2026-09-28T09:37:59.959232804Z",
      "endTime": "2026-09-28T10:02:05.043620822Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ma6360plq8ebh66osj03lihjgik still exists (deletion dispatched at 2026-09-28T09:38:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00290a2b8>: 
    resource group "rg-candidate-5-0-8h76gpwqntfk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-5z7vd6" in resourcegroup="rg-candidate-5-0-8h76gpwqntfk" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "name": "cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "status": "Failed",
      "startTime": "2026-09-28T09:35:00.890511025Z",
      "endTime": "2026-09-28T09:59:04.96675702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ao372vngnfec0pqf548ff4sjj still exists (deletion dispatched at 2026-09-28T09:35:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00290a2b8>: 
    resource group "rg-candidate-5-0-8h76gpwqntfk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-5z7vd6" in resourcegroup="rg-candidate-5-0-8h76gpwqntfk" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "name": "cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "status": "Failed",
      "startTime": "2026-09-28T09:35:00.890511025Z",
      "endTime": "2026-09-28T09:59:04.96675702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ao372vngnfec0pqf548ff4sjj still exists (deletion dispatched at 2026-09-28T09:35:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e523f0>: 
    resource group "v20261001preview-d4mgs8lq4npr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-d4mgs8lq4npr" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0792d6b3-7020-427d-84b3-8b07a05c3ad3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0792d6b3-7020-427d-84b3-8b07a05c3ad3",
      "name": "0792d6b3-7020-427d-84b3-8b07a05c3ad3",
      "status": "Failed",
      "startTime": "2026-09-28T09:27:03.486181728Z",
      "endTime": "2026-09-28T09:51:04.973500282Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u071ocvqmm5cqrijckm8sekgl81rd still exists (deletion dispatched at 2026-09-28T09:27:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e523f0>: 
    resource group "v20261001preview-d4mgs8lq4npr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-d4mgs8lq4npr" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0792d6b3-7020-427d-84b3-8b07a05c3ad3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0792d6b3-7020-427d-84b3-8b07a05c3ad3",
      "name": "0792d6b3-7020-427d-84b3-8b07a05c3ad3",
      "status": "Failed",
      "startTime": "2026-09-28T09:27:03.486181728Z",
      "endTime": "2026-09-28T09:51:04.973500282Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u071ocvqmm5cqrijckm8sekgl81rd still exists (deletion dispatched at 2026-09-28T09:27:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001700288>: 
    resource group "private-keyvault-vfws486bsfgl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-vfws486bsfgl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "name": "5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "status": "Failed",
      "startTime": "2026-09-28T09:29:57.435847003Z",
      "endTime": "2026-09-28T09:54:04.986151784Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u18j4sj43hhdfhhr4o61io72f8235 still exists (deletion dispatched at 2026-09-28T09:29:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001700288>: 
    resource group "private-keyvault-vfws486bsfgl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-vfws486bsfgl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "name": "5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "status": "Failed",
      "startTime": "2026-09-28T09:29:57.435847003Z",
      "endTime": "2026-09-28T09:54:04.986151784Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u18j4sj43hhdfhhr4o61io72f8235 still exists (deletion dispatched at 2026-09-28T09:29:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00186c0c0>: 
    resource group "rg-np-dg-minor-4jkg9h-xz9w7xmrt69c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-4jkg9h" in resourcegroup="rg-np-dg-minor-4jkg9h-xz9w7xmrt69c" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a3184ba-a34e-4171-90e1-59c285842841
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a3184ba-a34e-4171-90e1-59c285842841",
      "name": "2a3184ba-a34e-4171-90e1-59c285842841",
      "status": "Failed",
      "startTime": "2026-09-28T10:04:36.362367042Z",
      "endTime": "2026-09-28T10:28:40.099696412Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubaokiuhrcoo558a2ghta34iks995 still exists (deletion dispatched at 2026-09-28T10:04:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00186c0c0>: 
    resource group "rg-np-dg-minor-4jkg9h-xz9w7xmrt69c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-4jkg9h" in resourcegroup="rg-np-dg-minor-4jkg9h-xz9w7xmrt69c" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a3184ba-a34e-4171-90e1-59c285842841
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a3184ba-a34e-4171-90e1-59c285842841",
      "name": "2a3184ba-a34e-4171-90e1-59c285842841",
      "status": "Failed",
      "startTime": "2026-09-28T10:04:36.362367042Z",
      "endTime": "2026-09-28T10:28:40.099696412Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubaokiuhrcoo558a2ghta34iks995 still exists (deletion dispatched at 2026-09-28T10:04:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014aa0c0>: 
    resource group "rg-cp-ystream-upgrade-4-21-tbjnlwkt5shm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-89fffp" in resourcegroup="rg-cp-ystream-upgrade-4-21-tbjnlwkt5shm" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/661a4e43-2a8c-46d1-b23e-1fe0392ec81c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/661a4e43-2a8c-46d1-b23e-1fe0392ec81c",
      "name": "661a4e43-2a8c-46d1-b23e-1fe0392ec81c",
      "status": "Failed",
      "startTime": "2026-09-28T09:31:16.445346573Z",
      "endTime": "2026-09-28T09:55:24.93685083Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u09q1a1rcl3l0m7c7el9kbge10oe7 still exists (deletion dispatched at 2026-09-28T09:31:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014aa0c0>: 
    resource group "rg-cp-ystream-upgrade-4-21-tbjnlwkt5shm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-89fffp" in resourcegroup="rg-cp-ystream-upgrade-4-21-tbjnlwkt5shm" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/661a4e43-2a8c-46d1-b23e-1fe0392ec81c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/661a4e43-2a8c-46d1-b23e-1fe0392ec81c",
      "name": "661a4e43-2a8c-46d1-b23e-1fe0392ec81c",
      "status": "Failed",
      "startTime": "2026-09-28T09:31:16.445346573Z",
      "endTime": "2026-09-28T09:55:24.93685083Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u09q1a1rcl3l0m7c7el9kbge10oe7 still exists (deletion dispatched at 2026-09-28T09:31:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:299]: failed to update cluster with new KMS key with the second rotation
Unexpected error:
    <*fmt.wrapErrors | 0xc00123cdb0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sfghgk8d4rdx" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-sfghgk8d4rdx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:299]: failed to update cluster with new KMS key with the second rotation
Unexpected error:
    <*fmt.wrapErrors | 0xc00123cdb0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sfghgk8d4rdx" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-sfghgk8d4rdx, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002212048>: 
    resource group "e2e-cidr-connectivity-qw9smztvq7ws": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qw9smztvq7ws" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/304f8159-6c5e-4d11-8b14-b44d5bcb0d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/304f8159-6c5e-4d11-8b14-b44d5bcb0d51",
      "name": "304f8159-6c5e-4d11-8b14-b44d5bcb0d51",
      "status": "Failed",
      "startTime": "2026-09-28T09:38:13.479803853Z",
      "endTime": "2026-09-28T10:02:14.975908515Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u121rl1nqjoihtgis9hmujce91fc8 still exists (deletion dispatched at 2026-09-28T09:38:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002212048>: 
    resource group "e2e-cidr-connectivity-qw9smztvq7ws": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qw9smztvq7ws" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/304f8159-6c5e-4d11-8b14-b44d5bcb0d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/304f8159-6c5e-4d11-8b14-b44d5bcb0d51",
      "name": "304f8159-6c5e-4d11-8b14-b44d5bcb0d51",
      "status": "Failed",
      "startTime": "2026-09-28T09:38:13.479803853Z",
      "endTime": "2026-09-28T10:02:14.975908515Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u121rl1nqjoihtgis9hmujce91fc8 still exists (deletion dispatched at 2026-09-28T09:38:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002e96258>: 
    resource group "nodepool-update-nodes-pwhzr9kc5fmg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-pwhzr9kc5fmg" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c679bb39-251f-482e-aac7-5974edb2fbba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c679bb39-251f-482e-aac7-5974edb2fbba",
      "name": "c679bb39-251f-482e-aac7-5974edb2fbba",
      "status": "Failed",
      "startTime": "2026-09-28T09:48:09.967265057Z",
      "endTime": "2026-09-28T10:12:09.567399351Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u1fimurv5ioj3idg4r827ncmq2om6 still exists (deletion dispatched at 2026-09-28T09:48:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002e96258>: 
    resource group "nodepool-update-nodes-pwhzr9kc5fmg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-pwhzr9kc5fmg" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c679bb39-251f-482e-aac7-5974edb2fbba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c679bb39-251f-482e-aac7-5974edb2fbba",
      "name": "c679bb39-251f-482e-aac7-5974edb2fbba",
      "status": "Failed",
      "startTime": "2026-09-28T09:48:09.967265057Z",
      "endTime": "2026-09-28T10:12:09.567399351Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u1fimurv5ioj3idg4r827ncmq2om6 still exists (deletion dispatched at 2026-09-28T09:48:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000b82090>: 
    resource group "rg-zstream-upgrade-5-0-g9ntqxcmp6cx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-msh4hf" in resourcegroup="rg-zstream-upgrade-5-0-g9ntqxcmp6cx" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5007d355-1390-447c-a836-e7201fdc5f8f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5007d355-1390-447c-a836-e7201fdc5f8f",
      "name": "5007d355-1390-447c-a836-e7201fdc5f8f",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:21.326906807Z",
      "endTime": "2026-09-28T10:00:25.344782427Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0kbt737hlqiqbf1dq3vg54p17efo still exists (deletion dispatched at 2026-09-28T09:36:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000b82090>: 
    resource group "rg-zstream-upgrade-5-0-g9ntqxcmp6cx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-msh4hf" in resourcegroup="rg-zstream-upgrade-5-0-g9ntqxcmp6cx" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5007d355-1390-447c-a836-e7201fdc5f8f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5007d355-1390-447c-a836-e7201fdc5f8f",
      "name": "5007d355-1390-447c-a836-e7201fdc5f8f",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:21.326906807Z",
      "endTime": "2026-09-28T10:00:25.344782427Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0kbt737hlqiqbf1dq3vg54p17efo still exists (deletion dispatched at 2026-09-28T09:36:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000eb40f0>: 
    resource group "fips-enabled-bn5q2wwsf7rp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn5q2wwsf7rp" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "name": "3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:05.789845252Z",
      "endTime": "2026-09-28T09:54:14.850550256Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0hvkv6dq5otq5pkoot1lod893ptj still exists (deletion dispatched at 2026-09-28T09:30:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000eb40f0>: 
    resource group "fips-enabled-bn5q2wwsf7rp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn5q2wwsf7rp" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "name": "3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:05.789845252Z",
      "endTime": "2026-09-28T09:54:14.850550256Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0hvkv6dq5otq5pkoot1lod893ptj still exists (deletion dispatched at 2026-09-28T09:30:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0021100f0>: 
    resource group "rg-candidate-4-22-rsthdhx2w4z7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hqvm4n" in resourcegroup="rg-candidate-4-22-rsthdhx2w4z7" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf083a7d-366e-461c-b95d-686d145e886a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf083a7d-366e-461c-b95d-686d145e886a",
      "name": "cf083a7d-366e-461c-b95d-686d145e886a",
      "status": "Failed",
      "startTime": "2026-09-28T09:32:16.310177849Z",
      "endTime": "2026-09-28T09:56:25.119461348Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0o5ou7tq797vcmg5v755s0q2tngv still exists (deletion dispatched at 2026-09-28T09:32:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0021100f0>: 
    resource group "rg-candidate-4-22-rsthdhx2w4z7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hqvm4n" in resourcegroup="rg-candidate-4-22-rsthdhx2w4z7" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf083a7d-366e-461c-b95d-686d145e886a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf083a7d-366e-461c-b95d-686d145e886a",
      "name": "cf083a7d-366e-461c-b95d-686d145e886a",
      "status": "Failed",
      "startTime": "2026-09-28T09:32:16.310177849Z",
      "endTime": "2026-09-28T09:56:25.119461348Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0o5ou7tq797vcmg5v755s0q2tngv still exists (deletion dispatched at 2026-09-28T09:32:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fca138>: 
    resource group "rg-cordon-test-29mmt5q6w9zq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-29mmt5q6w9zq" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26f882f-0351-488a-bc80-3a658e3574f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26f882f-0351-488a-bc80-3a658e3574f5",
      "name": "d26f882f-0351-488a-bc80-3a658e3574f5",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:21.548016661Z",
      "endTime": "2026-09-28T09:54:24.966223651Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ojhe2srssobiigo2u2jk64kd72f still exists (deletion dispatched at 2026-09-28T09:30:26Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fca138>: 
    resource group "rg-cordon-test-29mmt5q6w9zq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-29mmt5q6w9zq" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26f882f-0351-488a-bc80-3a658e3574f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26f882f-0351-488a-bc80-3a658e3574f5",
      "name": "d26f882f-0351-488a-bc80-3a658e3574f5",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:21.548016661Z",
      "endTime": "2026-09-28T09:54:24.966223651Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ojhe2srssobiigo2u2jk64kd72f still exists (deletion dispatched at 2026-09-28T09:30:26Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00180c2e8>: 
    resource group "rg-negative-tests-rsx9k9dxrxnr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-rsx9k9dxrxnr" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1446725-d527-4a08-af82-4288738118c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1446725-d527-4a08-af82-4288738118c3",
      "name": "b1446725-d527-4a08-af82-4288738118c3",
      "status": "Failed",
      "startTime": "2026-09-28T09:43:15.547591968Z",
      "endTime": "2026-09-28T10:07:19.754648726Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubb559kv9knv31rsut3pvknqg6pg8 still exists (deletion dispatched at 2026-09-28T09:43:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00180c2e8>: 
    resource group "rg-negative-tests-rsx9k9dxrxnr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-rsx9k9dxrxnr" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1446725-d527-4a08-af82-4288738118c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1446725-d527-4a08-af82-4288738118c3",
      "name": "b1446725-d527-4a08-af82-4288738118c3",
      "status": "Failed",
      "startTime": "2026-09-28T09:43:15.547591968Z",
      "endTime": "2026-09-28T10:07:19.754648726Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubb559kv9knv31rsut3pvknqg6pg8 still exists (deletion dispatched at 2026-09-28T09:43:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002958120>: 
    resource group "rg-candidate-4-20-n6x2rfcpmr5t": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bkc8nt" in resourcegroup="rg-candidate-4-20-n6x2rfcpmr5t" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "name": "4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:44.717303002Z",
      "endTime": "2026-09-28T09:54:45.21106923Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ce2mv9aje3edhhstj5ffjhl0mpi still exists (deletion dispatched at 2026-09-28T09:31:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002958120>: 
    resource group "rg-candidate-4-20-n6x2rfcpmr5t": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bkc8nt" in resourcegroup="rg-candidate-4-20-n6x2rfcpmr5t" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "name": "4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:44.717303002Z",
      "endTime": "2026-09-28T09:54:45.21106923Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ce2mv9aje3edhhstj5ffjhl0mpi still exists (deletion dispatched at 2026-09-28T09:31:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00138e0d8>: 
    resource group "rg-np-downgrade-z882qs-gvknqpcq28h4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-z882qs" in resourcegroup="rg-np-downgrade-z882qs-gvknqpcq28h4" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caadaa7a-696b-4362-b133-bd079c0d1241
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caadaa7a-696b-4362-b133-bd079c0d1241",
      "name": "caadaa7a-696b-4362-b133-bd079c0d1241",
      "status": "Failed",
      "startTime": "2026-09-28T09:47:47.000063819Z",
      "endTime": "2026-09-28T10:11:49.934874142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubdmtg6c4vit45bipbt111i5ntkei still exists (deletion dispatched at 2026-09-28T09:47:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00138e0d8>: 
    resource group "rg-np-downgrade-z882qs-gvknqpcq28h4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-z882qs" in resourcegroup="rg-np-downgrade-z882qs-gvknqpcq28h4" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caadaa7a-696b-4362-b133-bd079c0d1241
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caadaa7a-696b-4362-b133-bd079c0d1241",
      "name": "caadaa7a-696b-4362-b133-bd079c0d1241",
      "status": "Failed",
      "startTime": "2026-09-28T09:47:47.000063819Z",
      "endTime": "2026-09-28T10:11:49.934874142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3ubdmtg6c4vit45bipbt111i5ntkei still exists (deletion dispatched at 2026-09-28T09:47:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011a84b0>: 
    resource group "rg-zstream-upgrade-4-22-wc2zsjqkql5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-22-pwm7lf" in resourcegroup="rg-zstream-upgrade-4-22-wc2zsjqkql5z" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b020a635-70f4-4fa3-bbeb-15a20c919dd5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b020a635-70f4-4fa3-bbeb-15a20c919dd5",
      "name": "b020a635-70f4-4fa3-bbeb-15a20c919dd5",
      "status": "Failed",
      "startTime": "2026-09-28T09:33:46.781992368Z",
      "endTime": "2026-09-28T09:57:54.928017678Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0gnt93pnblunkan391ssod5p70ip still exists (deletion dispatched at 2026-09-28T09:34:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011a84b0>: 
    resource group "rg-zstream-upgrade-4-22-wc2zsjqkql5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-22-pwm7lf" in resourcegroup="rg-zstream-upgrade-4-22-wc2zsjqkql5z" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b020a635-70f4-4fa3-bbeb-15a20c919dd5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b020a635-70f4-4fa3-bbeb-15a20c919dd5",
      "name": "b020a635-70f4-4fa3-bbeb-15a20c919dd5",
      "status": "Failed",
      "startTime": "2026-09-28T09:33:46.781992368Z",
      "endTime": "2026-09-28T09:57:54.928017678Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0gnt93pnblunkan391ssod5p70ip still exists (deletion dispatched at 2026-09-28T09:34:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010b4108>: 
    resource group "rg-zstream-upgrade-4-21-n75m8pfcjsx5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-kjvjjk" in resourcegroup="rg-zstream-upgrade-4-21-n75m8pfcjsx5" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/45231285-274d-43cd-98fa-1c2da5982b8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/45231285-274d-43cd-98fa-1c2da5982b8d",
      "name": "45231285-274d-43cd-98fa-1c2da5982b8d",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:34.341653398Z",
      "endTime": "2026-09-28T10:00:35.039725799Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0d23l1qdtpftb5407dlteafc6gfr still exists (deletion dispatched at 2026-09-28T09:36:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010b4108>: 
    resource group "rg-zstream-upgrade-4-21-n75m8pfcjsx5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-kjvjjk" in resourcegroup="rg-zstream-upgrade-4-21-n75m8pfcjsx5" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/45231285-274d-43cd-98fa-1c2da5982b8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/45231285-274d-43cd-98fa-1c2da5982b8d",
      "name": "45231285-274d-43cd-98fa-1c2da5982b8d",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:34.341653398Z",
      "endTime": "2026-09-28T10:00:35.039725799Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0d23l1qdtpftb5407dlteafc6gfr still exists (deletion dispatched at 2026-09-28T09:36:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001438840>: 
    resource group "rg-np-version-upgrade-gp7ddw-8fkxplz5m659": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-gp7ddw" in resourcegroup="rg-np-version-upgrade-gp7ddw-8fkxplz5m659" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/91fd8806-f596-45a2-8056-737eddeb0c8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/91fd8806-f596-45a2-8056-737eddeb0c8d",
      "name": "91fd8806-f596-45a2-8056-737eddeb0c8d",
      "status": "Failed",
      "startTime": "2026-09-28T09:43:50.208665116Z",
      "endTime": "2026-09-28T10:07:59.865594069Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u1fdl8l8ls86c304u4oqijmkekk18 still exists (deletion dispatched at 2026-09-28T09:43:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001438840>: 
    resource group "rg-np-version-upgrade-gp7ddw-8fkxplz5m659": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-gp7ddw" in resourcegroup="rg-np-version-upgrade-gp7ddw-8fkxplz5m659" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/91fd8806-f596-45a2-8056-737eddeb0c8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/91fd8806-f596-45a2-8056-737eddeb0c8d",
      "name": "91fd8806-f596-45a2-8056-737eddeb0c8d",
      "status": "Failed",
      "startTime": "2026-09-28T09:43:50.208665116Z",
      "endTime": "2026-09-28T10:07:59.865594069Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u1fdl8l8ls86c304u4oqijmkekk18 still exists (deletion dispatched at 2026-09-28T09:43:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a7e048>: 
    resource group "rg-cp-ystream-upgrade-5-0-q52nfkvp5d5w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-895w8v" in resourcegroup="rg-cp-ystream-upgrade-5-0-q52nfkvp5d5w" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/764e0c2e-ef4e-439a-97b4-27c1b0eec183
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/764e0c2e-ef4e-439a-97b4-27c1b0eec183",
      "name": "764e0c2e-ef4e-439a-97b4-27c1b0eec183",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:39.848641766Z",
      "endTime": "2026-09-28T10:00:44.979089844Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0jpa4etua5lae503qhk8taguuqgn still exists (deletion dispatched at 2026-09-28T09:36:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a7e048>: 
    resource group "rg-cp-ystream-upgrade-5-0-q52nfkvp5d5w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-895w8v" in resourcegroup="rg-cp-ystream-upgrade-5-0-q52nfkvp5d5w" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/764e0c2e-ef4e-439a-97b4-27c1b0eec183
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/764e0c2e-ef4e-439a-97b4-27c1b0eec183",
      "name": "764e0c2e-ef4e-439a-97b4-27c1b0eec183",
      "status": "Failed",
      "startTime": "2026-09-28T09:36:39.848641766Z",
      "endTime": "2026-09-28T10:00:44.979089844Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0jpa4etua5lae503qhk8taguuqgn still exists (deletion dispatched at 2026-09-28T09:36:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a02240>: 
    resource group "rg-np-version-upgrade-scljgc-q96m77tgzfb6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-scljgc" in resourcegroup="rg-np-version-upgrade-scljgc-q96m77tgzfb6" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1103c4f-e224-4b48-b9f0-0980b852bf71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1103c4f-e224-4b48-b9f0-0980b852bf71",
      "name": "d1103c4f-e224-4b48-b9f0-0980b852bf71",
      "status": "Failed",
      "startTime": "2026-09-28T09:42:29.944025Z",
      "endTime": "2026-09-28T10:06:29.757432329Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0onupvo7hktmjvhi336mbs8af0cq still exists (deletion dispatched at 2026-09-28T09:43:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a02240>: 
    resource group "rg-np-version-upgrade-scljgc-q96m77tgzfb6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-scljgc" in resourcegroup="rg-np-version-upgrade-scljgc-q96m77tgzfb6" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1103c4f-e224-4b48-b9f0-0980b852bf71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d1103c4f-e224-4b48-b9f0-0980b852bf71",
      "name": "d1103c4f-e224-4b48-b9f0-0980b852bf71",
      "status": "Failed",
      "startTime": "2026-09-28T09:42:29.944025Z",
      "endTime": "2026-09-28T10:06:29.757432329Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0onupvo7hktmjvhi336mbs8af0cq still exists (deletion dispatched at 2026-09-28T09:43:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015461e0>: 
    resource group "rg-candidate-4-23-wmbvtjdttxwg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-jt55tk" in resourcegroup="rg-candidate-4-23-wmbvtjdttxwg" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8af0bb-8a34-470d-bea3-06838637e747
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8af0bb-8a34-470d-bea3-06838637e747",
      "name": "ae8af0bb-8a34-470d-bea3-06838637e747",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:46.693994314Z",
      "endTime": "2026-09-28T09:54:55.540074205Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0k121hdg8khh203rm14pi2dgi08b still exists (deletion dispatched at 2026-09-28T09:30:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015461e0>: 
    resource group "rg-candidate-4-23-wmbvtjdttxwg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-jt55tk" in resourcegroup="rg-candidate-4-23-wmbvtjdttxwg" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8af0bb-8a34-470d-bea3-06838637e747
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8af0bb-8a34-470d-bea3-06838637e747",
      "name": "ae8af0bb-8a34-470d-bea3-06838637e747",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:46.693994314Z",
      "endTime": "2026-09-28T09:54:55.540074205Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0k121hdg8khh203rm14pi2dgi08b still exists (deletion dispatched at 2026-09-28T09:30:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:43:43Z
  Ended:       2026-09-28T09:49:43Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1041152-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1041152-svc).

alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:12:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z8g6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:15:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:15:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:16:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:17:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z8g6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 16 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="cloud-network-config-controller-58dfb59986-bsjtm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2b4fc680-6a59-448e-b315-f78694553d9a"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller-58dfb59986-bsjtm (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="cloud-network-config-controller-69985fdf59-fmd5d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="387b4461-aa28-4131-85f9-f2b77c4e2e0e"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/cloud-network-config-controller-69985fdf59-fmd5d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="cloud-network-config-controller-69985fdf59-fmd5d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="387b4461-aa28-4131-85f9-f2b77c4e2e0e"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/cloud-network-config-controller-69985fdf59-fmd5d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="csi-snapshot-controller-d4d6cd4f4-d2tqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c3f26a0c-8e66-4f75-bb2e-f2645071d798"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/csi-snapshot-controller-d4d6cd4f4-d2tqr (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="csi-snapshot-controller-d4d6cd4f4-d2tqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c3f26a0c-8e66-4f75-bb2e-f2645071d798"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/csi-snapshot-controller-d4d6cd4f4-d2tqr (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="csi-snapshot-controller-c9cf48cfd-vfz96", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e625ddc5-e22f-4d5c-bb2f-d4cb602de924"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/csi-snapshot-controller-c9cf48cfd-vfz96 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="cloud-network-config-controller-5c87577d75-pzm4d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f7312b5e-d9af-4928-8d9a-ea34f1380f1e"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller-5c87577d75-pzm4d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="cloud-network-config-controller-5b9c45d5bc-245tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e7400d-4c21-4a6d-a049-fcf41ebbe075"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller-5b9c45d5bc-245tg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="cloud-network-config-controller-58dfb59986-bsjtm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2b4fc680-6a59-448e-b315-f78694553d9a"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller-58dfb59986-bsjtm (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="csi-snapshot-controller-69d89876cf-6p78f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a54f46c-0195-4d5e-aae8-68c97f711f84"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/csi-snapshot-controller-69d89876cf-6p78f (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="cloud-network-config-controller-5c87577d75-pzm4d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f7312b5e-d9af-4928-8d9a-ea34f1380f1e"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller-5c87577d75-pzm4d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="csi-snapshot-controller-69d89876cf-6p78f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a54f46c-0195-4d5e-aae8-68c97f711f84"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/csi-snapshot-controller-69d89876cf-6p78f (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="csi-snapshot-controller-7d6c8c6fbb-xmtc5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07ef49a6-b3c7-427b-ab6f-44204ea64710"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/csi-snapshot-controller-7d6c8c6fbb-xmtc5 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="csi-snapshot-controller-c9cf48cfd-vfz96", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e625ddc5-e22f-4d5c-bb2f-d4cb602de924"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/csi-snapshot-controller-c9cf48cfd-vfz96 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="cloud-network-config-controller-5b9c45d5bc-245tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e7400d-4c21-4a6d-a049-fcf41ebbe075"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller-5b9c45d5bc-245tg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="csi-snapshot-controller-7d6c8c6fbb-xmtc5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07ef49a6-b3c7-427b-ab6f-44204ea64710"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/csi-snapshot-controller-7d6c8c6fbb-xmtc5 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:36:38Z
  Ended:       2026-09-28T09:42:38Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j1041152-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3u0ioqrfjsn6acu7pjpg12lkcddspk-i4m7g3h8t9f6r1s", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j1041152-mgmt-2
Namespace: ocm-arohcpci01-2t3u0ioqrfjsn6acu7pjpg12lkcddspk-i4m7g3h8t9f6r1s

Firing 2:
  State:       Resolved
  Started:     2026-09-28T10:21:38Z
  Ended:       2026-09-28T10:28:38Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j1041152-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3u0ma6360plq8ebh66osj03lihjgik-j3g2n6z5l7j6d3s", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j1041152-mgmt-1
Namespace: ocm-arohcpci01-2t3u0ma6360plq8ebh66osj03lihjgik-j3g2n6z5l7j6d3s

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:24:44Z
  Ended:       2026-09-28T10:37:42Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j1041152-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3ubdmtg6c4vit45bipbt111i5ntkei-m5v3h7g8t9d4c9v", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j1041152-mgmt-1
Namespace: ocm-arohcpci01-2t3ubdmtg6c4vit45bipbt111i5ntkei-m5v3h7g8t9d4c9v

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:24:42Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j1041152-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3u0ma6360plq8ebh66osj03lihjgik-j3g2n6z5l7j6d3s", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j1041152-mgmt-1
Namespace: ocm-arohcpci01-2t3u0ma6360plq8ebh66osj03lihjgik-j3g2n6z5l7j6d3s

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T10:21:38Z
  Ended:       2026-09-28T10:28:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j1041152-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3ubdmtg6c4vit45bipbt111i5ntkei-m5v3h7g8t9d4c9v", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j1041152-mgmt-1
Namespace: ocm-arohcpci01-2t3ubdmtg6c4vit45bipbt111i5ntkei-m5v3h7g8t9d4c9v

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:00:41Z
  Ended:       2026-09-28T09:37:40Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j3076352-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3totsphp1krc605sa3296gqttbfa03-l4x6j4k8k1d0k5l", region="centralus", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j3076352-mgmt-1
Namespace: ocm-arohcpci01-2t3totsphp1krc605sa3296gqttbfa03-l4x6j4k8k1d0k5l

Firing 2:
  State:       Resolved
  Started:     2026-09-28T09:02:40Z
  Ended:       2026-09-28T09:11:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j3076352-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3toqtlnbhe7378oukj6o7c79jqcnoj-b6u2o5n7c3y1y3q", region="centralus", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j3076352-mgmt-2
Namespace: ocm-arohcpci01-2t3toqtlnbhe7378oukj6o7c79jqcnoj-b6u2o5n7c3y1y3q

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:03:45Z
  Ended:       2026-09-28T10:02:42Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j3076352-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3totsphp1krc605sa3296gqttbfa03-l4x6j4k8k1d0k5l", region="centralus", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j3076352-mgmt-1
Namespace: ocm-arohcpci01-2t3totsphp1krc605sa3296gqttbfa03-l4x6j4k8k1d0k5l

Firing 2:
  State:       Resolved
  Started:     2026-09-28T09:05:46Z
  Ended:       2026-09-28T09:35:42Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j3076352-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3toqtlnbhe7378oukj6o7c79jqcnoj-b6u2o5n7c3y1y3q", region="centralus", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j3076352-mgmt-2
Namespace: ocm-arohcpci01-2t3toqtlnbhe7378oukj6o7c79jqcnoj-b6u2o5n7c3y1y3q

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:32:29Z
  Ended:       2026-09-28T09:38:24Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0371968-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0371968-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T09:18:31Z
  Ended:       2026-09-28T09:30:33Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8274688-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8274688-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T08:45:52Z
  Ended:       2026-09-28T08:51:55Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8234240-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8234240-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T08:07:37Z
  Ended:       2026-09-28T08:55:37Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j6653824-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3stml6c3gc3s82ndnpr0dtik24as4p-q2l2i3u5g4g6x0r", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j6653824-mgmt-2
Namespace: ocm-arohcpci01-2t3stml6c3gc3s82ndnpr0dtik24as4p-q2l2i3u5g4g6x0r

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T08:04:37Z
  Ended:       2026-09-28T08:30:38Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j6653824-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3stml6c3gc3s82ndnpr0dtik24as4p-q2l2i3u5g4g6x0r", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j6653824-mgmt-2
Namespace: ocm-arohcpci01-2t3stml6c3gc3s82ndnpr0dtik24as4p-q2l2i3u5g4g6x0r

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T08:14:08Z
  Ended:       2026-09-28T08:38:09Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j2651264-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3t2a1queqj7uhi7vm5f3spuqd988nl-private-kas", region="centralus", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2651264-mgmt-2
Namespace: ocm-arohcpci01-2t3t2a1queqj7uhi7vm5f3spuqd988nl-private-kas

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T08:12:21Z
  Ended:       2026-09-28T08:44:23Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2651264-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2651264-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T08:17:15Z
  Ended:       2026-09-28T09:03:09Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j2651264-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3t2a1queqj7uhi7vm5f3spuqd988nl-private-kas", region="centralus", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2651264-mgmt-2
Namespace: ocm-arohcpci01-2t3t2a1queqj7uhi7vm5f3spuqd988nl-private-kas

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-rxbdcb" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001090b40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-rxbdcb: failed waiting for cluster="cp-ystream-upgrade-4-23-rxbdcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-jvcp8dmnbrt7" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "name": "1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:56.829077997Z",
      "endTime": "2026-09-28T07:47:03.158495671Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud failed: Get \"https://api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud:443/healthz?verbose\": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-rxbdcb" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001090b40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-rxbdcb: failed waiting for cluster="cp-ystream-upgrade-4-23-rxbdcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-jvcp8dmnbrt7" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "name": "1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:56.829077997Z",
      "endTime": "2026-09-28T07:47:03.158495671Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud failed: Get \"https://api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud:443/healthz?verbose\": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019421c8>: 
    resource group "rg-candidate-5-0-z7h95fzkng4w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-tvrknh" in resourcegroup="rg-candidate-5-0-z7h95fzkng4w" to finish deleting: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/acf5d502-002a-40c9-939d-0b196b8376ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/acf5d502-002a-40c9-939d-0b196b8376ff",
      "name": "acf5d502-002a-40c9-939d-0b196b8376ff",
      "status": "Failed",
      "startTime": "2026-09-28T07:42:39.657365818Z",
      "endTime": "2026-09-28T08:06:43.09244758Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3sd9jqifdo5nbri2gfug3fjubjauaa still exists (deletion dispatched at 2026-09-28T07:42:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019421c8>: 
    resource group "rg-candidate-5-0-z7h95fzkng4w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-tvrknh" in resourcegroup="rg-candidate-5-0-z7h95fzkng4w" to finish deleting: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/acf5d502-002a-40c9-939d-0b196b8376ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/acf5d502-002a-40c9-939d-0b196b8376ff",
      "name": "acf5d502-002a-40c9-939d-0b196b8376ff",
      "status": "Failed",
      "startTime": "2026-09-28T07:42:39.657365818Z",
      "endTime": "2026-09-28T08:06:43.09244758Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3sd9jqifdo5nbri2gfug3fjubjauaa still exists (deletion dispatched at 2026-09-28T07:42:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001138408>: 
    resource group "rg-candidate-4-23-thxgh9fqwrb4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-8jxpm2" in resourcegroup="rg-candidate-4-23-thxgh9fqwrb4" to finish deleting: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/140bfc0f-802a-403b-ad5f-d4afa52fb8f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/140bfc0f-802a-403b-ad5f-d4afa52fb8f7",
      "name": "140bfc0f-802a-403b-ad5f-d4afa52fb8f7",
      "status": "Failed",
      "startTime": "2026-09-28T07:40:12.927522807Z",
      "endTime": "2026-09-28T08:04:13.132621699Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3sd9i7p3emdffa2gd3ri3elbvmi10h still exists (deletion dispatched at 2026-09-28T07:40:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001138408>: 
    resource group "rg-candidate-4-23-thxgh9fqwrb4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-8jxpm2" in resourcegroup="rg-candidate-4-23-thxgh9fqwrb4" to finish deleting: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/140bfc0f-802a-403b-ad5f-d4afa52fb8f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/140bfc0f-802a-403b-ad5f-d4afa52fb8f7",
      "name": "140bfc0f-802a-403b-ad5f-d4afa52fb8f7",
      "status": "Failed",
      "startTime": "2026-09-28T07:40:12.927522807Z",
      "endTime": "2026-09-28T08:04:13.132621699Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3sd9i7p3emdffa2gd3ri3elbvmi10h still exists (deletion dispatched at 2026-09-28T07:40:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0017a4198>: 
    resource group "private-kas-c65n2fbgmgnm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-c65n2fbgmgnm" to finish deleting: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d639e17-fc2d-4fbd-b7d8-4049afd23792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d639e17-fc2d-4fbd-b7d8-4049afd23792",
      "name": "8d639e17-fc2d-4fbd-b7d8-4049afd23792",
      "status": "Failed",
      "startTime": "2026-09-28T07:41:12.899620144Z",
      "endTime": "2026-09-28T08:05:13.036001959Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3sct197en3e7rl4tferai9as2qo90u still exists (deletion dispatched at 2026-09-28T07:41:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0017a4198>: 
    resource group "private-kas-c65n2fbgmgnm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-c65n2fbgmgnm" to finish deleting: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d639e17-fc2d-4fbd-b7d8-4049afd23792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d639e17-fc2d-4fbd-b7d8-4049afd23792",
      "name": "8d639e17-fc2d-4fbd-b7d8-4049afd23792",
      "status": "Failed",
      "startTime": "2026-09-28T07:41:12.899620144Z",
      "endTime": "2026-09-28T08:05:13.036001959Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3sct197en3e7rl4tferai9as2qo90u still exists (deletion dispatched at 2026-09-28T07:41:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000911660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6qt82pv5cb4k" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088",
      "name": "cd024c78-818b-4b49-8634-2f704d9a8088",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:28.191649845Z",
      "endTime": "2026-09-28T07:47:33.272794005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000911660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6qt82pv5cb4k" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088",
      "name": "cd024c78-818b-4b49-8634-2f704d9a8088",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:28.191649845Z",
      "endTime": "2026-09-28T07:47:33.272794005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--hspw4fkzkwjz/cluster-hs-pre-nhlgdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc000eb3280>: 
    failed to create HCP cluster cluster-hs-pre-nhlgdc: failed waiting for cluster="cluster-hs-pre-nhlgdc" in resourcegroup="rg-hs-pre--hspw4fkzkwjz" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e30d74a4-97b0-45b3-bb34-c0689a890bb7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e30d74a4-97b0-45b3-bb34-c0689a890bb7",
      "name": "e30d74a4-97b0-45b3-bb34-c0689a890bb7",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:58.926212691Z",
      "endTime": "2026-09-28T07:47:03.131231516Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m5s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--hspw4fkzkwjz/cluster-hs-pre-nhlgdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc000eb3280>: 
    failed to create HCP cluster cluster-hs-pre-nhlgdc: failed waiting for cluster="cluster-hs-pre-nhlgdc" in resourcegroup="rg-hs-pre--hspw4fkzkwjz" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e30d74a4-97b0-45b3-bb34-c0689a890bb7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e30d74a4-97b0-45b3-bb34-c0689a890bb7",
      "name": "e30d74a4-97b0-45b3-bb34-c0689a890bb7",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:58.926212691Z",
      "endTime": "2026-09-28T07:47:03.131231516Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m5s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000b72f40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-pqfhb9gph6n6" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1554388-8ac1-4e37-a154-657d9e219a9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1554388-8ac1-4e37-a154-657d9e219a9c",
      "name": "e1554388-8ac1-4e37-a154-657d9e219a9c",
      "status": "Failed",
      "startTime": "2026-09-28T07:33:16.16764174Z",
      "endTime": "2026-09-28T07:52:22.952806599Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-dl2qp-qdb4g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-pqfhb9gph6n6--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dl2qp-qdb4g to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-dl2qp-qdb4g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-pqfhb9gph6n6--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dl2qp-qdb4g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000b72f40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-pqfhb9gph6n6" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1554388-8ac1-4e37-a154-657d9e219a9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1554388-8ac1-4e37-a154-657d9e219a9c",
      "name": "e1554388-8ac1-4e37-a154-657d9e219a9c",
      "status": "Failed",
      "startTime": "2026-09-28T07:33:16.16764174Z",
      "endTime": "2026-09-28T07:52:22.952806599Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-dl2qp-qdb4g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-pqfhb9gph6n6--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dl2qp-qdb4g to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-dl2qp-qdb4g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-pqfhb9gph6n6--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dl2qp-qdb4g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0015a0780>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-5p54bqs5s2wq" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1573c01-1be7-4a7a-a184-70dcc6eb1000
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1573c01-1be7-4a7a-a184-70dcc6eb1000",
      "name": "b1573c01-1be7-4a7a-a184-70dcc6eb1000",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:48.290879335Z",
      "endTime": "2026-09-28T07:47:53.160718992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0015a0780>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-5p54bqs5s2wq" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1573c01-1be7-4a7a-a184-70dcc6eb1000
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1573c01-1be7-4a7a-a184-70dcc6eb1000",
      "name": "b1573c01-1be7-4a7a-a184-70dcc6eb1000",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:48.290879335Z",
      "endTime": "2026-09-28T07:47:53.160718992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000fda740>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dsp8rmtwf8ld" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70235f5c-fb03-427a-9ea4-ad4fabb543b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70235f5c-fb03-427a-9ea4-ad4fabb543b4",
      "name": "70235f5c-fb03-427a-9ea4-ad4fabb543b4",
      "status": "Failed",
      "startTime": "2026-09-28T07:19:31.516544795Z",
      "endTime": "2026-09-28T07:48:33.353465776Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 24m12s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000fda740>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dsp8rmtwf8ld" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70235f5c-fb03-427a-9ea4-ad4fabb543b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70235f5c-fb03-427a-9ea4-ad4fabb543b4",
      "name": "70235f5c-fb03-427a-9ea4-ad4fabb543b4",
      "status": "Failed",
      "startTime": "2026-09-28T07:19:31.516544795Z",
      "endTime": "2026-09-28T07:48:33.353465776Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 24m12s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T07:32:17Z
  Ended:       2026-09-28T08:32:17Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3sd8fa4sfo5kgkevh0op6a88e9j5ei-z7j8g5y8a5a3b8k", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3sd8fa4sfo5kgkevh0op6a88e9j5ei-z7j8g5y8a5a3b8k

Firing 2:
  State:       Resolved
  Started:     2026-09-28T07:35:16Z
  Ended:       2026-09-28T08:07:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h

Firing 3:
  State:       Resolved
  Started:     2026-09-28T07:37:17Z
  Ended:       2026-09-28T08:07:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4259200-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-2
Namespace: ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le

Firing 4:
  State:       Resolved
  Started:     2026-09-28T07:37:17Z
  Ended:       2026-09-28T07:57:16Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3sd354a3bjr7bte08phb8vflqovkof-acr-pull-create", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3sd354a3bjr7bte08phb8vflqovkof-acr-pull-create

Firing 5:
  State:       Resolved
  Started:     2026-09-28T07:43:19Z
  Ended:       2026-09-28T08:13:17Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4259200-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-2
Namespace: ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T07:30:18Z
  Ended:       2026-09-28T07:54:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t", pod="router-74979554f7-8crr2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t/router-74979554f7-8crr2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T07:31:16Z
  Ended:       2026-09-28T07:56:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster", pod="router-7b9fff456-6vss4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster/router-7b9fff456-6vss4 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T07:31:16Z
  Ended:       2026-09-28T07:56:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster", pod="router-57cc58f4cd-7msjr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster/router-57cc58f4cd-7msjr has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T07:33:18Z
  Ended:       2026-09-28T07:56:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4259200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r", pod="router-7698497b4b-2px6r", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r/router-7698497b4b-2px6r has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T07:33:19Z
  Ended:       2026-09-28T07:56:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4259200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r", pod="router-7698497b4b-87nkt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r/router-7698497b4b-87nkt has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T07:33:17Z
  Ended:       2026-09-28T07:42:17Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h

Firing 2:
  State:       Resolved
  Started:     2026-09-28T07:34:21Z
  Ended:       2026-09-28T07:42:17Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j4259200-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-2
Namespace: ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le

Firing 3:
  State:       Resolved
  Started:     2026-09-28T07:39:16Z
  Ended:       2026-09-28T07:49:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3sd15tp1vsqeia30g4p4aldq8urook-i5j0s3k4w4m1r6u", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3sd15tp1vsqeia30g4p4aldq8urook-i5j0s3k4w4m1r6u

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T07:36:16Z
  Ended:       2026-09-28T08:07:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h

Firing 2:
  State:       Resolved
  Started:     2026-09-28T07:37:17Z
  Ended:       2026-09-28T07:44:20Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j4259200-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-2
Namespace: ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le

Firing 3:
  State:       Resolved
  Started:     2026-09-28T07:42:17Z
  Ended:       2026-09-28T08:14:17Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3sd15tp1vsqeia30g4p4aldq8urook-i5j0s3k4w4m1r6u", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3sd15tp1vsqeia30g4p4aldq8urook-i5j0s3k4w4m1r6u

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T07:53:06Z
  Ended:       2026-09-28T07:59:07Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4259200-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4259200-svc).

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T07:29:17Z
  Ended:       2026-09-28T08:07:16Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3sd8fa4sfo5kgkevh0op6a88e9j5ei-z7j8g5y8a5a3b8k", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3sd8fa4sfo5kgkevh0op6a88e9j5ei-z7j8g5y8a5a3b8k

Firing 2:
  State:       Resolved
  Started:     2026-09-28T07:32:17Z
  Ended:       2026-09-28T07:42:15Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3scp4va5n5nn2ra64ruq0e3s68195j-u1z0s4v7m7d0a6h

Firing 3:
  State:       Resolved
  Started:     2026-09-28T07:34:16Z
  Ended:       2026-09-28T07:48:15Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4259200-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3sd354a3bjr7bte08phb8vflqovkof-acr-pull-create", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-1
Namespace: ocm-arohcpci01-2t3sd354a3bjr7bte08phb8vflqovkof-acr-pull-create

Firing 4:
  State:       Resolved
  Started:     2026-09-28T07:34:16Z
  Ended:       2026-09-28T07:44:16Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4259200-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-2
Namespace: ocm-arohcpci01-2t3sdcaqi76dbq451pj95v1uqnfr208u-ingress-cert-le

Firing 5:
  State:       Resolved
  Started:     2026-09-28T07:40:15Z
  Ended:       2026-09-28T07:50:15Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4259200-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4259200-mgmt-2
Namespace: ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster

alert fired 16 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:26:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:26:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:26:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:26:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:29:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.233:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z54j2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:29:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.233:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z54j2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:29:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.233:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z54j2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:29:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.233:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z54j2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdh60bmkrgtngq1o0rh0b4nlqksl7-d4l4a8z0v3y5r6r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sdmetkr1pjassdmepvbt8femrrbda-cilium-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3se1b6bje1ulsarhngi9asa53lodqr-agg-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-28T08:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4259200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4259200/providers/microsoft.monitor/accounts/hcps-j4259200", namespace="ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-f4z5m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3sd9oqcm1kefsb7dqks9ar62agsjku-e7g4a1t9d1d9r8t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b2f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-rxccfwgxq7jj" to finish creating: GET https://rp.j1154560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ba1066e-9913-4a3a-ab91-5b893d5447b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ba1066e-9913-4a3a-ab91-5b893d5447b0",
      "name": "7ba1066e-9913-4a3a-ab91-5b893d5447b0",
      "status": "Failed",
      "startTime": "2026-09-28T06:32:25.841883321Z",
      "endTime": "2026-09-28T06:51:27.808098854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b6o4b6r8b3p4s9g-np-1-s8xp9-hrvhd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b6o4b6r8b3p4s9g-np-1-s8xp9-fsxfs: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-rxccfwgxq7jj--managed/b6o4b6r8b3p4s9g-np-1-s8xp9-fsxfs-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rxccfwgxq7jj--managed/providers/Microsoft.Network/networkInterfaces/b6o4b6r8b3p4s9g-np-1-s8xp9-fsxfs-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to http... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b2f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-rxccfwgxq7jj" to finish creating: GET https://rp.j1154560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ba1066e-9913-4a3a-ab91-5b893d5447b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ba1066e-9913-4a3a-ab91-5b893d5447b0",
      "name": "7ba1066e-9913-4a3a-ab91-5b893d5447b0",
      "status": "Failed",
      "startTime": "2026-09-28T06:32:25.841883321Z",
      "endTime": "2026-09-28T06:51:27.808098854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b6o4b6r8b3p4s9g-np-1-s8xp9-hrvhd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b6o4b6r8b3p4s9g-np-1-s8xp9-fsxfs: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-rxccfwgxq7jj--managed/b6o4b6r8b3p4s9g-np-1-s8xp9-fsxfs-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rxccfwgxq7jj--managed/providers/Microsoft.Network/networkInterfaces/b6o4b6r8b3p4s9g-np-1-s8xp9-fsxfs-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to http... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:58:33Z
  Ended:       2026-09-28T07:06:36Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1154560-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1154560-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:51:50Z
  Ended:       2026-09-28T06:57:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1154560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3rl3f8tj7aka16mnasqurrb5ebbvn0-w7g1w7z1p7g5a8x", pod="kube-apiserver-8448d69f6f-jqnnb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3rl3f8tj7aka16mnasqurrb5ebbvn0-w7g1w7z1p7g5a8x/kube-apiserver-8448d69f6f-jqnnb has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:57:58Z
  Ended:       2026-09-28T07:09:57Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1195392-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1195392-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001303da0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-n7hm7qphnd7k" to finish creating: GET https://rp.j3362944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14660fe0-ba9c-4568-a549-06eba5f1111c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14660fe0-ba9c-4568-a549-06eba5f1111c",
      "name": "14660fe0-ba9c-4568-a549-06eba5f1111c",
      "status": "Failed",
      "startTime": "2026-09-28T06:35:59.37107386Z",
      "endTime": "2026-09-28T06:55:00.67957674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-hhqxx-z5glq: Failed: virtualmachine failed to create or update. err: failed to get existing resource complex-cilium-kv-n7hm7qphnd7k--managed/cilium-cluster-cilium-np-hhqxx-z5glq (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-n7hm7qphnd7k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-hhqxx-z5glq\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3d8c36e1-ae7a-42ef-b7d1-ea4667708d30' exceeded. Please try again after '10' seconds after additional tokens are ava... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001303da0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-n7hm7qphnd7k" to finish creating: GET https://rp.j3362944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14660fe0-ba9c-4568-a549-06eba5f1111c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/14660fe0-ba9c-4568-a549-06eba5f1111c",
      "name": "14660fe0-ba9c-4568-a549-06eba5f1111c",
      "status": "Failed",
      "startTime": "2026-09-28T06:35:59.37107386Z",
      "endTime": "2026-09-28T06:55:00.67957674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-hhqxx-z5glq: Failed: virtualmachine failed to create or update. err: failed to get existing resource complex-cilium-kv-n7hm7qphnd7k--managed/cilium-cluster-cilium-np-hhqxx-z5glq (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-n7hm7qphnd7k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-hhqxx-z5glq\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3d8c36e1-ae7a-42ef-b7d1-ea4667708d30' exceeded. Please try again after '10' seconds after additional tokens are ava... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000ad61a0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-nfzkdr" in resourcegroup="rg-np-downgrade-nfzkdr-5bwdcqfbzpwz" to finish creating: GET https://rp.j4189312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca540d48-e274-44eb-b0a7-ddf6f21af0cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca540d48-e274-44eb-b0a7-ddf6f21af0cc",
      "name": "ca540d48-e274-44eb-b0a7-ddf6f21af0cc",
      "status": "Failed",
      "startTime": "2026-09-28T06:59:50.075602846Z",
      "endTime": "2026-09-28T07:18:52.768957656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-nfzkdr-5bwdcqfbzpwz-np-dg-nfzkdr--managed/providers/Microsoft.Compute/virtualMachines/i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-nfzkdr-5bwdcqfbzpwz-np-dg-nfzkdr--managed/providers/Microsoft.Compute/virtualMachines/i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000ad61a0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-nfzkdr" in resourcegroup="rg-np-downgrade-nfzkdr-5bwdcqfbzpwz" to finish creating: GET https://rp.j4189312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca540d48-e274-44eb-b0a7-ddf6f21af0cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca540d48-e274-44eb-b0a7-ddf6f21af0cc",
      "name": "ca540d48-e274-44eb-b0a7-ddf6f21af0cc",
      "status": "Failed",
      "startTime": "2026-09-28T06:59:50.075602846Z",
      "endTime": "2026-09-28T07:18:52.768957656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-nfzkdr-5bwdcqfbzpwz-np-dg-nfzkdr--managed/providers/Microsoft.Compute/virtualMachines/i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-nfzkdr-5bwdcqfbzpwz-np-dg-nfzkdr--managed/providers/Microsoft.Compute/virtualMachines/i2k1v9u6g0j4j1a-npdg-4-21-tbbgs-mxb9k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T07:01:37Z
  Ended:       2026-09-28T07:07:34Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4189312-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4189312-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:58:35Z
  Ended:       2026-09-28T07:06:36Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2978944-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2978944-svc).

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:52:56Z
  Ended:       2026-09-28T07:21:54Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j2978944-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3riui9072n9eok39gpkgl46upeo7m9-private-kas", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2978944-mgmt-2
Namespace: ocm-arohcpci01-2t3riui9072n9eok39gpkgl46upeo7m9-private-kas

Firing 2:
  State:       Resolved
  Started:     2026-09-28T07:37:54Z
  Ended:       2026-09-28T07:45:54Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j2978944-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3rsi6td6p6c241sbjaqid23velc7j6-j6u3g7r8x8f0v5c", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2978944-mgmt-1
Namespace: ocm-arohcpci01-2t3rsi6td6p6c241sbjaqid23velc7j6-j6u3g7r8x8f0v5c

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:49:57Z
  Ended:       2026-09-28T06:56:55Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2978944-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3riui9072n9eok39gpkgl46upeo7m9-private-kas", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2978944-mgmt-2
Namespace: ocm-arohcpci01-2t3riui9072n9eok39gpkgl46upeo7m9-private-kas

Firing 2:
  State:       Resolved
  Started:     2026-09-28T07:34:53Z
  Ended:       2026-09-28T07:42:55Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2978944-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3rsi6td6p6c241sbjaqid23velc7j6-j6u3g7r8x8f0v5c", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2978944-mgmt-1
Namespace: ocm-arohcpci01-2t3rsi6td6p6c241sbjaqid23velc7j6-j6u3g7r8x8f0v5c

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-x7z7b7"
Unexpected error:
    <*fmt.wrapError | 0xc000bab020>: 
    DNS for route host agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-x7z7b7"
Unexpected error:
    <*fmt.wrapError | 0xc000bab020>: 
    DNS for route host agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:12:52Z
  Ended:       2026-09-28T06:19:52Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7422208-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7422208-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000874f60>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-14/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000874f60>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-14/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000c3c580>: 
    DNS for route host agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000c3c580>: 
    DNS for route host agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cf1400>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9wpvql" in resourcegroup="rg-np-version-upgrade-9wpvql-69cnxf97b854" to finish creating: GET https://rp.j9776640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5e618a3-8d2b-42aa-9601-b69b45f47871
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5e618a3-8d2b-42aa-9601-b69b45f47871",
      "name": "a5e618a3-8d2b-42aa-9601-b69b45f47871",
      "status": "Failed",
      "startTime": "2026-09-28T05:29:42.522112187Z",
      "endTime": "2026-09-28T05:48:46.557258592Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9wpvql-69cnxf97b854-np-upgrade-9w-04ede2a4/providers/Microsoft.Compute/virtualMachines/n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9wpvql-69cnxf97b854-np-upgrade-9w-04ede2a4/providers/Microsoft.Compute/virtualMachines/n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cf1400>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9wpvql" in resourcegroup="rg-np-version-upgrade-9wpvql-69cnxf97b854" to finish creating: GET https://rp.j9776640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5e618a3-8d2b-42aa-9601-b69b45f47871
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5e618a3-8d2b-42aa-9601-b69b45f47871",
      "name": "a5e618a3-8d2b-42aa-9601-b69b45f47871",
      "status": "Failed",
      "startTime": "2026-09-28T05:29:42.522112187Z",
      "endTime": "2026-09-28T05:48:46.557258592Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9wpvql-69cnxf97b854-np-upgrade-9w-04ede2a4/providers/Microsoft.Compute/virtualMachines/n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9wpvql-69cnxf97b854-np-upgrade-9w-04ede2a4/providers/Microsoft.Compute/virtualMachines/n1h7x2o1d8f9m4k-npupgrade-4-20-chfrz-jbjzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:53:31Z
  Ended:       2026-09-28T05:59:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9776640-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9776640-svc).

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001f902d0>: 
    resource group "rg-candidate-4-23-9gw25wvwtcq8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-h9898s" in resourcegroup="rg-candidate-4-23-9gw25wvwtcq8" to finish deleting: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f64f2b5-a8be-4840-8a0f-0ab62f2387b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f64f2b5-a8be-4840-8a0f-0ab62f2387b2",
      "name": "9f64f2b5-a8be-4840-8a0f-0ab62f2387b2",
      "status": "Failed",
      "startTime": "2026-09-28T05:19:21.358341405Z",
      "endTime": "2026-09-28T05:43:24.1869414Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3q9mk4rs0jgf269qvh2civ1ghcvp1q still exists (deletion dispatched at 2026-09-28T05:19:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001f902d0>: 
    resource group "rg-candidate-4-23-9gw25wvwtcq8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-h9898s" in resourcegroup="rg-candidate-4-23-9gw25wvwtcq8" to finish deleting: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f64f2b5-a8be-4840-8a0f-0ab62f2387b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f64f2b5-a8be-4840-8a0f-0ab62f2387b2",
      "name": "9f64f2b5-a8be-4840-8a0f-0ab62f2387b2",
      "status": "Failed",
      "startTime": "2026-09-28T05:19:21.358341405Z",
      "endTime": "2026-09-28T05:43:24.1869414Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3q9mk4rs0jgf269qvh2civ1ghcvp1q still exists (deletion dispatched at 2026-09-28T05:19:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:164 with:
expected HTTP 200 from sample app via internal LB, got 503
Expected
    <string>: ...
to equal
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:164 with:
expected HTTP 200 from sample app via internal LB, got 503
Expected
    <string>: ...
to equal
    <string>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000fc65a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-6vnwh8vjnz8r" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f",
      "name": "0b3163de-a897-4f45-9b77-e785bdfab09f",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.798033495Z",
      "endTime": "2026-09-28T05:22:54.610455796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000fc65a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-6vnwh8vjnz8r" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f",
      "name": "0b3163de-a897-4f45-9b77-e785bdfab09f",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.798033495Z",
      "endTime": "2026-09-28T05:22:54.610455796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ae31a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-5cbtfzkp497x" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "name": "6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:33.784163345Z",
      "endTime": "2026-09-28T05:22:34.354425424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ae31a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-5cbtfzkp497x" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "name": "6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:33.784163345Z",
      "endTime": "2026-09-28T05:22:34.354425424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cb0840>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rbwvmrnzw8j5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e",
      "name": "5bdae3db-e612-4f27-8693-02200538e08e",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:23.74994885Z",
      "endTime": "2026-09-28T05:23:24.026384001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cb0840>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rbwvmrnzw8j5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e",
      "name": "5bdae3db-e612-4f27-8693-02200538e08e",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:23.74994885Z",
      "endTime": "2026-09-28T05:23:24.026384001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0005d2ce0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sbtnmq6dk9ch" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758",
      "name": "1f13de10-f443-42eb-b391-6417a9194758",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:46.663436186Z",
      "endTime": "2026-09-28T05:24:54.296612042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0005d2ce0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sbtnmq6dk9ch" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758",
      "name": "1f13de10-f443-42eb-b391-6417a9194758",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:46.663436186Z",
      "endTime": "2026-09-28T05:24:54.296612042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc0010c41e0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-jfwb2dgmj26t" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "name": "a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:17.858913168Z",
      "endTime": "2026-09-28T05:23:24.118546984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc0010c41e0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-jfwb2dgmj26t" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "name": "a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:17.858913168Z",
      "endTime": "2026-09-28T05:23:24.118546984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0004e23d8>: 
    resource group "private-keyvault-7rld8zcz29zp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-7rld8zcz29zp" to finish deleting: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4041230b-c16b-483c-b3c9-78fc4b2a05c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4041230b-c16b-483c-b3c9-78fc4b2a05c6",
      "name": "4041230b-c16b-483c-b3c9-78fc4b2a05c6",
      "status": "Failed",
      "startTime": "2026-09-28T05:16:11.06337557Z",
      "endTime": "2026-09-28T05:40:14.206723479Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3qacjj77p1dno3e8ufhl76fug7bvbc still exists (deletion dispatched at 2026-09-28T05:16:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0004e23d8>: 
    resource group "private-keyvault-7rld8zcz29zp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-7rld8zcz29zp" to finish deleting: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4041230b-c16b-483c-b3c9-78fc4b2a05c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4041230b-c16b-483c-b3c9-78fc4b2a05c6",
      "name": "4041230b-c16b-483c-b3c9-78fc4b2a05c6",
      "status": "Failed",
      "startTime": "2026-09-28T05:16:11.06337557Z",
      "endTime": "2026-09-28T05:40:14.206723479Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3qacjj77p1dno3e8ufhl76fug7bvbc still exists (deletion dispatched at 2026-09-28T05:16:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2s722b
Unexpected error:
    <*fmt.wrapError | 0xc000bfeb60>: 
    failed to create HCP cluster np-noedge-2s722b: failed waiting for cluster="np-noedge-2s722b" in resourcegroup="rg-np-noedge-2s722b-ttwnzxstsj4p" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93",
      "name": "18b3752b-dfe7-413c-83e0-e4570361ec93",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:09.735885483Z",
      "endTime": "2026-09-28T05:23:14.561838226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2s722b
Unexpected error:
    <*fmt.wrapError | 0xc000bfeb60>: 
    failed to create HCP cluster np-noedge-2s722b: failed waiting for cluster="np-noedge-2s722b" in resourcegroup="rg-np-noedge-2s722b-ttwnzxstsj4p" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93",
      "name": "18b3752b-dfe7-413c-83e0-e4570361ec93",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:09.735885483Z",
      "endTime": "2026-09-28T05:23:14.561838226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00106c320>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-qdqkz9rfmldx" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23",
      "name": "0c723202-68f5-41c7-a5d4-762157d31d23",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.997312592Z",
      "endTime": "2026-09-28T05:22:54.143428969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00106c320>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-qdqkz9rfmldx" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23",
      "name": "0c723202-68f5-41c7-a5d4-762157d31d23",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.997312592Z",
      "endTime": "2026-09-28T05:22:54.143428969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6kj9nw5mjrc2/cluster-hs-pre-fvbpxs should provision
Unexpected error:
    <*fmt.wrapError | 0xc000db85e0>: 
    failed to create HCP cluster cluster-hs-pre-fvbpxs: failed waiting for cluster="cluster-hs-pre-fvbpxs" in resourcegroup="rg-hs-pre--6kj9nw5mjrc2" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd",
      "name": "2f78ab67-468f-449c-890b-25423489ffcd",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.674200437Z",
      "endTime": "2026-09-28T05:23:04.406727689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6kj9nw5mjrc2/cluster-hs-pre-fvbpxs should provision
Unexpected error:
    <*fmt.wrapError | 0xc000db85e0>: 
    failed to create HCP cluster cluster-hs-pre-fvbpxs: failed waiting for cluster="cluster-hs-pre-fvbpxs" in resourcegroup="rg-hs-pre--6kj9nw5mjrc2" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd",
      "name": "2f78ab67-468f-449c-890b-25423489ffcd",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.674200437Z",
      "endTime": "2026-09-28T05:23:04.406727689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f0f2c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p82fzl2z5vk5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022",
      "name": "03f41a1e-bd33-4df1-9f47-56824bb74022",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:14.362679822Z",
      "endTime": "2026-09-28T05:24:15.510395217Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f0f2c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p82fzl2z5vk5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022",
      "name": "03f41a1e-bd33-4df1-9f47-56824bb74022",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:14.362679822Z",
      "endTime": "2026-09-28T05:24:15.510395217Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gng5crkc9wh8/cluster-candidate-4-22-w8wr72 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00117a200>: 
    failed to create HCP cluster cluster-candidate-4-22-w8wr72: failed waiting for cluster="cluster-candidate-4-22-w8wr72" in resourcegroup="rg-candidate-4-22-gng5crkc9wh8" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "name": "00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.970731237Z",
      "endTime": "2026-09-28T05:23:04.188577598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, router; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gng5crkc9wh8/cluster-candidate-4-22-w8wr72 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00117a200>: 
    failed to create HCP cluster cluster-candidate-4-22-w8wr72: failed waiting for cluster="cluster-candidate-4-22-w8wr72" in resourcegroup="rg-candidate-4-22-gng5crkc9wh8" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "name": "00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.970731237Z",
      "endTime": "2026-09-28T05:23:04.188577598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, router; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc001904960>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vhq6gmmfrw9m" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vhq6gmmfrw9m, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc001904960>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vhq6gmmfrw9m" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vhq6gmmfrw9m, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001698220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hqr7q7xq4llr" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "name": "e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:08.942854859Z",
      "endTime": "2026-09-28T05:23:14.16072165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001698220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hqr7q7xq4llr" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "name": "e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:08.942854859Z",
      "endTime": "2026-09-28T05:23:14.16072165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:03:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:03:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:04:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:06:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T06:06:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/hcps-j4374016", namespace="ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:01:43Z
  Ended:       2026-09-28T05:34:42Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4374016-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4374016-svc).

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:05:43Z
  Ended:       2026-09-28T05:33:44Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4374016-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4374016-mgmt-2
Namespace: ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001

Firing 2:
  State:       Resolved
  Started:     2026-09-28T05:08:45Z
  Ended:       2026-09-28T05:34:44Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4374016-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4374016-mgmt-1
Namespace: ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y

Firing 3:
  State:       Resolved
  Started:     2026-09-28T05:09:44Z
  Ended:       2026-09-28T05:15:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4374016-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3q9v1bm9djo4ed6huo3q8b2cid8e8d-s4a1j7z9i0s2s7m", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4374016-mgmt-1
Namespace: ocm-arohcpci01-2t3q9v1bm9djo4ed6huo3q8b2cid8e8d-s4a1j7z9i0s2s7m

Firing 4:
  State:       Resolved
  Started:     2026-09-28T05:11:44Z
  Ended:       2026-09-28T05:19:44Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4374016-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3q9ljvg6g8v7atf8e4uj1uvv1o29vp-z3b4r7r7o7d8c5l", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4374016-mgmt-2
Namespace: ocm-arohcpci01-2t3q9ljvg6g8v7atf8e4uj1uvv1o29vp-z3b4r7r7o7d8c5l

Firing 5:
  State:       Resolved
  Started:     2026-09-28T05:12:46Z
  Ended:       2026-09-28T05:22:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4374016-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3q9md15na81g26g32hk4dt01vu985b-private-kas", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4374016-mgmt-1
Namespace: ocm-arohcpci01-2t3q9md15na81g26g32hk4dt01vu985b-private-kas

Firing 6:
  State:       Resolved
  Started:     2026-09-28T05:17:42Z
  Ended:       2026-09-28T05:31:46Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j4374016-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3qa7kalf0nmi953o4ju9ljktm9ed87-i1j4h0b1t2z2z3r", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4374016-mgmt-2
Namespace: ocm-arohcpci01-2t3qa7kalf0nmi953o4ju9ljktm9ed87-i1j4h0b1t2z2z3r

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:12:47Z
  Ended:       2026-09-28T05:57:45Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j4374016-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4374016-mgmt-1
Namespace: ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:09:45Z
  Ended:       2026-09-28T05:32:47Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j4374016-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j4374016-mgmt-1
Namespace: ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y

alert fired 22 time(s) (18 unknown, 4 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:06:51Z
  Ended:       2026-09-28T05:32:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster", pod="router-7f5bb6dd68-mjsm7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster/router-7f5bb6dd68-mjsm7 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T05:06:53Z
  Ended:       2026-09-28T05:32:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster", pod="router-7f5bb6dd68-9v56k", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3qakt150613r65up3s1cocvavjoksi-agg-cluster/router-7f5bb6dd68-9v56k has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T05:06:53Z
  Ended:       2026-09-28T05:31:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9r4v4h8ssovlhfggk1nbpf0t4f5t-b2s4u1s0i0q3z0q", pod="router-79b6987566-srrmw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9r4v4h8ssovlhfggk1nbpf0t4f5t-b2s4u1s0i0q3z0q/router-79b6987566-srrmw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:30:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9sbeie1jf1oga9bmqauhj98o2lm6-z7q3j8s4e3j3u3b", pod="router-d8ccb4d9c-4bmnv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9sbeie1jf1oga9bmqauhj98o2lm6-z7q3j8s4e3j3u3b/router-d8ccb4d9c-4bmnv has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:30:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9sbeie1jf1oga9bmqauhj98o2lm6-z7q3j8s4e3j3u3b", pod="router-d8ccb4d9c-7kk78", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9sbeie1jf1oga9bmqauhj98o2lm6-z7q3j8s4e3j3u3b/router-d8ccb4d9c-7kk78 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:32:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster", pod="router-78f9bfbb8f-rkcnj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster/router-78f9bfbb8f-rkcnj has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:30:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001", pod="router-6fc8bdc65f-ldlt9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001/router-6fc8bdc65f-ldlt9 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:31:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y", pod="router-7f659c45d8-5255k", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y/router-7f659c45d8-5255k has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:30:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001", pod="router-6fc8bdc65f-zhdpc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001/router-6fc8bdc65f-zhdpc has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:32:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster", pod="router-78f9bfbb8f-h658c", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster/router-78f9bfbb8f-h658c has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:31:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y", pod="router-7f659c45d8-d5t8j", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y/router-7f659c45d8-d5t8j has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-28T05:07:56Z
  Ended:       2026-09-28T05:32:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster", pod="router-78f9bfbb8f-txt79", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3qaci6g1hrolig0jqqqo4o0ii7hd2r-cilium-cluster/router-78f9bfbb8f-txt79 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-28T05:09:53Z
  Ended:       2026-09-28T05:16:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9mk4rs0jgf269qvh2civ1ghcvp1q-q7r1o0f1f9v9t9w", pod="azure-disk-csi-driver-controller-5cdd6fb8fd-qb979", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9mk4rs0jgf269qvh2civ1ghcvp1q-q7r1o0f1f9v9t9w/azure-disk-csi-driver-controller-5cdd6fb8fd-qb979 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-28T05:12:52Z
  Ended:       2026-09-28T05:21:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9utda6pi90u9tf0cjvsofdad0ucs-i1i3q3s8s2n4k3q", pod="ignition-server-proxy-6b5d65465c-9c7v8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9utda6pi90u9tf0cjvsofdad0ucs-i1i3q3s8s2n4k3q/ignition-server-proxy-6b5d65465c-9c7v8 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-28T05:12:52Z
  Ended:       2026-09-28T05:18:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9md15na81g26g32hk4dt01vu985b-private-kas", pod="ignition-server-proxy-5c4dd747d9-vm4gw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9md15na81g26g32hk4dt01vu985b-private-kas/ignition-server-proxy-5c4dd747d9-vm4gw has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-28T05:12:52Z
  Ended:       2026-09-28T05:18:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9md15na81g26g32hk4dt01vu985b-private-kas", pod="kube-controller-manager-6b788bd4b6-hnwq2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9md15na81g26g32hk4dt01vu985b-private-kas/kube-controller-manager-6b788bd4b6-hnwq2 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-28T05:12:52Z
  Ended:       2026-09-28T05:18:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9utda6pi90u9tf0cjvsofdad0ucs-i1i3q3s8s2n4k3q", pod="kube-controller-manager-658dbd6bb8-prjzm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9utda6pi90u9tf0cjvsofdad0ucs-i1i3q3s8s2n4k3q/kube-controller-manager-658dbd6bb8-prjzm has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-28T05:30:55Z
  Ended:       2026-09-28T05:36:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3q9q7agfjgjsadrgs66grkjrrebpou-ephemeral-disk", pod="openshift-apiserver-788cc5f496-6jkkq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3q9q7agfjgjsadrgs66grkjrrebpou-ephemeral-disk/openshift-apiserver-788cc5f496-6jkkq has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:30:55Z
  Ended:       2026-09-28T05:36:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/services-j4374016", namespace="klusterlet-2t3qakt150613r65up3s1cocvavjoksi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t3qakt150613r65up3s1cocvavjoksi/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T05:30:55Z
  Ended:       2026-09-28T05:36:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/services-j4374016", namespace="klusterlet-2t3qakt150613r65up3s1cocvavjoksi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t3qakt150613r65up3s1cocvavjoksi/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:08:50Z
  Ended:       2026-09-28T05:58:45Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4374016-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4374016-mgmt-2
Namespace: ocm-arohcpci01-2t3q9oqqnica740m7cbrv4g0ui9okfrv-v20261001

Firing 2:
  State:       Resolved
  Started:     2026-09-28T05:14:45Z
  Ended:       2026-09-28T05:44:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4374016-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3q9ljvg6g8v7atf8e4uj1uvv1o29vp-z3b4r7r7o7d8c5l", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4374016-mgmt-2
Namespace: ocm-arohcpci01-2t3q9ljvg6g8v7atf8e4uj1uvv1o29vp-z3b4r7r7o7d8c5l

Firing 3:
  State:       Resolved
  Started:     2026-09-28T05:15:43Z
  Ended:       2026-09-28T05:59:48Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4374016-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4374016-mgmt-1
Namespace: ocm-arohcpci01-2t3q9r41nmns94dh4sm3puhs32psv67j-m1v4i4a9a2m8b6y

Firing 4:
  State:       Resolved
  Started:     2026-09-28T05:20:44Z
  Ended:       2026-09-28T05:53:46Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j4374016-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3qa7kalf0nmi953o4ju9ljktm9ed87-i1j4h0b1t2z2z3r", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j4374016-mgmt-2
Namespace: ocm-arohcpci01-2t3qa7kalf0nmi953o4ju9ljktm9ed87-i1j4h0b1t2z2z3r

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc0033b6750>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-nh4vk2wgfp5z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-nh4vk2wgfp5z, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc0033b6750>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-nh4vk2wgfp5z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-nh4vk2wgfp5z, error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T03:43:51Z
  Ended:       2026-09-28T03:49:48Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j1071360-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3p00agfmo4aa4d1u3cphgepq43dhn2-v20261001", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j1071360-mgmt-2
Namespace: ocm-arohcpci01-2t3p00agfmo4aa4d1u3cphgepq43dhn2-v20261001

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012c4150>: 
    resource group "complex-cilium-kv-glbv5phs24ph": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-glbv5phs24ph" to finish deleting: GET https://rp.j9759360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75cb0f96-c479-4ace-abc8-ee149e772174
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75cb0f96-c479-4ace-abc8-ee149e772174",
      "name": "75cb0f96-c479-4ace-abc8-ee149e772174",
      "status": "Failed",
      "startTime": "2026-09-28T02:31:25.384468319Z",
      "endTime": "2026-09-28T02:55:30.931830528Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3nqnt80g6r52fu96h7a5rhdsivrefe still exists (deletion dispatched at 2026-09-28T02:31:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012c4150>: 
    resource group "complex-cilium-kv-glbv5phs24ph": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-glbv5phs24ph" to finish deleting: GET https://rp.j9759360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75cb0f96-c479-4ace-abc8-ee149e772174
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75cb0f96-c479-4ace-abc8-ee149e772174",
      "name": "75cb0f96-c479-4ace-abc8-ee149e772174",
      "status": "Failed",
      "startTime": "2026-09-28T02:31:25.384468319Z",
      "endTime": "2026-09-28T02:55:30.931830528Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3nqnt80g6r52fu96h7a5rhdsivrefe still exists (deletion dispatched at 2026-09-28T02:31:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc0005607a0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-555cz2" in resourcegroup="rg-np-downgrade-555cz2-rdlncxrfkhtc" to finish creating: GET https://rp.j9759360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc685bb9-f2bf-4194-a9fc-20acce3ce179
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc685bb9-f2bf-4194-a9fc-20acce3ce179",
      "name": "fc685bb9-f2bf-4194-a9fc-20acce3ce179",
      "status": "Failed",
      "startTime": "2026-09-28T02:35:28.867850368Z",
      "endTime": "2026-09-28T02:54:30.899062424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-555cz2-rdlncxrfkhtc-np-dg-555cz2--managed/providers/Microsoft.Compute/virtualMachines/w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-555cz2-rdlncxrfkhtc-np-dg-555cz2--managed/providers/Microsoft.Compute/virtualMachines/w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc0005607a0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-555cz2" in resourcegroup="rg-np-downgrade-555cz2-rdlncxrfkhtc" to finish creating: GET https://rp.j9759360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc685bb9-f2bf-4194-a9fc-20acce3ce179
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc685bb9-f2bf-4194-a9fc-20acce3ce179",
      "name": "fc685bb9-f2bf-4194-a9fc-20acce3ce179",
      "status": "Failed",
      "startTime": "2026-09-28T02:35:28.867850368Z",
      "endTime": "2026-09-28T02:54:30.899062424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-555cz2-rdlncxrfkhtc-np-dg-555cz2--managed/providers/Microsoft.Compute/virtualMachines/w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-555cz2-rdlncxrfkhtc-np-dg-555cz2--managed/providers/Microsoft.Compute/virtualMachines/w3a0i0z2r1p2g0g-npdg-4-21-pslgq-rs7b4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T02:39:49Z
  Ended:       2026-09-28T02:45:46Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9759360-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9759360-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0001585e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-zknh6gjcfdvf" to finish creating: GET https://rp.j2553344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c774cfb-cf99-4fb2-a28d-fbbb15de258f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c774cfb-cf99-4fb2-a28d-fbbb15de258f",
      "name": "3c774cfb-cf99-4fb2-a28d-fbbb15de258f",
      "status": "Failed",
      "startTime": "2026-09-28T02:15:44.28328608Z",
      "endTime": "2026-09-28T02:34:53.308480382Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-zknh6gjcfdvf--managed/providers/Microsoft.Compute/virtualMachines/u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-zknh6gjcfdvf--managed/providers/Microsoft.Compute/virtualMachines/u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0001585e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-zknh6gjcfdvf" to finish creating: GET https://rp.j2553344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c774cfb-cf99-4fb2-a28d-fbbb15de258f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3c774cfb-cf99-4fb2-a28d-fbbb15de258f",
      "name": "3c774cfb-cf99-4fb2-a28d-fbbb15de258f",
      "status": "Failed",
      "startTime": "2026-09-28T02:15:44.28328608Z",
      "endTime": "2026-09-28T02:34:53.308480382Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-zknh6gjcfdvf--managed/providers/Microsoft.Compute/virtualMachines/u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-zknh6gjcfdvf--managed/providers/Microsoft.Compute/virtualMachines/u9y5r9m7v3u4f0e-np-1-hw27d-4ns2q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00085bca0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-8vl8f9l9j29x" to finish creating: GET https://rp.j9353728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e5850d6-dc29-4e45-835c-6b9898d7a6dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e5850d6-dc29-4e45-835c-6b9898d7a6dc",
      "name": "3e5850d6-dc29-4e45-835c-6b9898d7a6dc",
      "status": "Failed",
      "startTime": "2026-09-28T01:19:59.863031878Z",
      "endTime": "2026-09-28T01:39:01.927768441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-8vl8f9l9j29x--managed/providers/Microsoft.Compute/virtualMachines/d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-8vl8f9l9j29x--managed/providers/Microsoft.Compute/virtualMachines/d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00085bca0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-8vl8f9l9j29x" to finish creating: GET https://rp.j9353728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e5850d6-dc29-4e45-835c-6b9898d7a6dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e5850d6-dc29-4e45-835c-6b9898d7a6dc",
      "name": "3e5850d6-dc29-4e45-835c-6b9898d7a6dc",
      "status": "Failed",
      "startTime": "2026-09-28T01:19:59.863031878Z",
      "endTime": "2026-09-28T01:39:01.927768441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-8vl8f9l9j29x--managed/providers/Microsoft.Compute/virtualMachines/d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-8vl8f9l9j29x--managed/providers/Microsoft.Compute/virtualMachines/d3p4u4b1l4h8m6c-np-1-bkt5k-rknzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:46:36Z
  Ended:       2026-09-28T01:54:53Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9353728-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9353728-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:57:15Z
  Ended:       2026-09-28T02:20:15Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j9353728-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3n2c15frlnm47g6kiq5c2e1cd6hcrh-agg-cluster", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j9353728-mgmt-2
Namespace: ocm-arohcpci01-2t3n2c15frlnm47g6kiq5c2e1cd6hcrh-agg-cluster

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:27:15Z
  Ended:       2026-09-28T01:35:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5263104-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5263104-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002728c00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-542tb2cfkqw8" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-542tb2cfkqw8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002728c00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-542tb2cfkqw8" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-542tb2cfkqw8, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc003167da0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kg9lzmtgpr42" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kg9lzmtgpr42, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc003167da0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kg9lzmtgpr42" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kg9lzmtgpr42, error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:14:56Z
  Ended:       2026-09-28T01:33:00Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3086080-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:29:52Z
  Ended:       2026-09-28T01:40:51Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3086080-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3086080-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004c7460>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-vrnhsnxhvbfn" to finish creating: GET https://rp.j3407872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eaa0085b-c669-46fa-ae33-9cc3f5616168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eaa0085b-c669-46fa-ae33-9cc3f5616168",
      "name": "eaa0085b-c669-46fa-ae33-9cc3f5616168",
      "status": "Failed",
      "startTime": "2026-09-28T01:05:43.045871667Z",
      "endTime": "2026-09-28T01:24:48.358531583Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e7c4q2e4p3k7u1k-np-1-8882r-8hcd9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-8hcd9 to exist\nMachine e7c4q2e4p3k7u1k-np-1-8882r-r4rnq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-r4rnq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e7c4q2e4p3k7u1k-np-1-8882r-8hcd9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-8hcd9 to exist\nMachine e7c4q2e4p3k7u1k-np-1-8882r-r4rnq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-r4rnq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004c7460>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-vrnhsnxhvbfn" to finish creating: GET https://rp.j3407872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eaa0085b-c669-46fa-ae33-9cc3f5616168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eaa0085b-c669-46fa-ae33-9cc3f5616168",
      "name": "eaa0085b-c669-46fa-ae33-9cc3f5616168",
      "status": "Failed",
      "startTime": "2026-09-28T01:05:43.045871667Z",
      "endTime": "2026-09-28T01:24:48.358531583Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e7c4q2e4p3k7u1k-np-1-8882r-8hcd9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-8hcd9 to exist\nMachine e7c4q2e4p3k7u1k-np-1-8882r-r4rnq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-r4rnq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e7c4q2e4p3k7u1k-np-1-8882r-8hcd9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-8hcd9 to exist\nMachine e7c4q2e4p3k7u1k-np-1-8882r-r4rnq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vrnhsnxhvbfn--managed/providers/Microsoft.Compute/virtualMachines/e7c4q2e4p3k7u1k-np-1-8882r-r4rnq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:02:29Z
  Ended:       2026-09-28T01:17:31Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j3407872-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3mqa3q05v2kb8qpc4fuhqft0q3g7f2-s7y7f0s8w4l4q7o", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j3407872-mgmt-1
Namespace: ocm-arohcpci01-2t3mqa3q05v2kb8qpc4fuhqft0q3g7f2-s7y7f0s8w4l4q7o

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:28:29Z
  Ended:       2026-09-28T01:34:32Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3407872-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3407872-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:05:30Z
  Ended:       2026-09-28T01:42:31Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j3407872-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3mqa3q05v2kb8qpc4fuhqft0q3g7f2-s7y7f0s8w4l4q7o", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j3407872-mgmt-1
Namespace: ocm-arohcpci01-2t3mqa3q05v2kb8qpc4fuhqft0q3g7f2-s7y7f0s8w4l4q7o

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:06:00Z
  Ended:       2026-09-28T01:26:57Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2994560-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3mnh30341mbni3cq0802m7mt0jj8ii-x2a5l3m8n0n6f6h", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2994560-mgmt-2
Namespace: ocm-arohcpci01-2t3mnh30341mbni3cq0802m7mt0jj8ii-x2a5l3m8n0n6f6h

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T01:08:59Z
  Ended:       2026-09-28T01:28:10Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j2994560-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3mnh30341mbni3cq0802m7mt0jj8ii-x2a5l3m8n0n6f6h", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2994560-mgmt-2
Namespace: ocm-arohcpci01-2t3mnh30341mbni3cq0802m7mt0jj8ii-x2a5l3m8n0n6f6h

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001806078>: 
    resource group "admin-credential-lifecycle-test-2mncmbxff48k": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-gzn4lc" in resourcegroup="admin-credential-lifecycle-test-2mncmbxff48k" to finish deleting: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/100c82ed-e364-4103-9093-3cc047de25dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/100c82ed-e364-4103-9093-3cc047de25dc",
      "name": "100c82ed-e364-4103-9093-3cc047de25dc",
      "status": "Failed",
      "startTime": "2026-09-28T01:00:46.173719966Z",
      "endTime": "2026-09-28T01:24:48.870398522Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3mm6bjna4825qvibg4tmtpm7c95ke0 still exists (deletion dispatched at 2026-09-28T01:00:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001806078>: 
    resource group "admin-credential-lifecycle-test-2mncmbxff48k": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-gzn4lc" in resourcegroup="admin-credential-lifecycle-test-2mncmbxff48k" to finish deleting: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/100c82ed-e364-4103-9093-3cc047de25dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/100c82ed-e364-4103-9093-3cc047de25dc",
      "name": "100c82ed-e364-4103-9093-3cc047de25dc",
      "status": "Failed",
      "startTime": "2026-09-28T01:00:46.173719966Z",
      "endTime": "2026-09-28T01:24:48.870398522Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3mm6bjna4825qvibg4tmtpm7c95ke0 still exists (deletion dispatched at 2026-09-28T01:00:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000ca85b8>: 
    resource group "admin-api-serialconsole-st5xdz2xcc6m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-st5xdz2xcc6m" to finish deleting: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d9aeeb2-65e9-44c7-90ba-68e9e692244c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d9aeeb2-65e9-44c7-90ba-68e9e692244c",
      "name": "3d9aeeb2-65e9-44c7-90ba-68e9e692244c",
      "status": "Failed",
      "startTime": "2026-09-28T01:06:48.076247478Z",
      "endTime": "2026-09-28T01:30:48.731406356Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3mmeectlf6lq0ip66b1ckuumq6c71t still exists (deletion dispatched at 2026-09-28T01:06:49Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000ca85b8>: 
    resource group "admin-api-serialconsole-st5xdz2xcc6m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-st5xdz2xcc6m" to finish deleting: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d9aeeb2-65e9-44c7-90ba-68e9e692244c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d9aeeb2-65e9-44c7-90ba-68e9e692244c",
      "name": "3d9aeeb2-65e9-44c7-90ba-68e9e692244c",
      "status": "Failed",
      "startTime": "2026-09-28T01:06:48.076247478Z",
      "endTime": "2026-09-28T01:30:48.731406356Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3mmeectlf6lq0ip66b1ckuumq6c71t still exists (deletion dispatched at 2026-09-28T01:06:49Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014796a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-qq55pmfqfj46" to finish creating: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1b5c158-195f-45c4-8138-3df46250e606
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1b5c158-195f-45c4-8138-3df46250e606",
      "name": "e1b5c158-195f-45c4-8138-3df46250e606",
      "status": "Failed",
      "startTime": "2026-09-28T01:10:15.13644965Z",
      "endTime": "2026-09-28T01:29:18.755471449Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-qq55pmfqfj46--managed/providers/Microsoft.Compute/virtualMachines/h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-qq55pmfqfj46--managed/providers/Microsoft.Compute/virtualMachines/h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014796a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-qq55pmfqfj46" to finish creating: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1b5c158-195f-45c4-8138-3df46250e606
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1b5c158-195f-45c4-8138-3df46250e606",
      "name": "e1b5c158-195f-45c4-8138-3df46250e606",
      "status": "Failed",
      "startTime": "2026-09-28T01:10:15.13644965Z",
      "endTime": "2026-09-28T01:29:18.755471449Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-qq55pmfqfj46--managed/providers/Microsoft.Compute/virtualMachines/h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-qq55pmfqfj46--managed/providers/Microsoft.Compute/virtualMachines/h2y7r3j2i6e7b9v-autoscale-noaz-kch6d-5xtbd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001948060>: 
    resource group "customer-rg-hx6m7trvhcw8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-hx6m7trvhcw8" to finish deleting: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/010ae2fe-5ec0-4b79-bfe4-03112aab36d3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/010ae2fe-5ec0-4b79-bfe4-03112aab36d3",
      "name": "010ae2fe-5ec0-4b79-bfe4-03112aab36d3",
      "status": "Failed",
      "startTime": "2026-09-28T00:59:43.756752481Z",
      "endTime": "2026-09-28T01:23:48.800910697Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3mmkbf6c7cpr22sjfg901rr6tj3ald still exists (deletion dispatched at 2026-09-28T00:59:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001948060>: 
    resource group "customer-rg-hx6m7trvhcw8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-hx6m7trvhcw8" to finish deleting: GET https://rp.j4479872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/010ae2fe-5ec0-4b79-bfe4-03112aab36d3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/010ae2fe-5ec0-4b79-bfe4-03112aab36d3",
      "name": "010ae2fe-5ec0-4b79-bfe4-03112aab36d3",
      "status": "Failed",
      "startTime": "2026-09-28T00:59:43.756752481Z",
      "endTime": "2026-09-28T01:23:48.800910697Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3mmkbf6c7cpr22sjfg901rr6tj3ald still exists (deletion dispatched at 2026-09-28T00:59:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb"
Unexpected error:
    <*fmt.wrapError | 0xc00024f2c0>: 
    failed to create NodePool np-fnv7lb: failed waiting for nodepool="np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb" in resourcegroup="rg-candidate-5-0-v6nvmt4rxzjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a",
      "name": "203ad465-1e1f-4337-9659-f5fe2972a72a",
      "status": "Failed",
      "startTime": "2026-09-28T21:17:06.138034027Z",
      "endTime": "2026-09-28T21:36:06.809868093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb"
Unexpected error:
    <*fmt.wrapError | 0xc00024f2c0>: 
    failed to create NodePool np-fnv7lb: failed waiting for nodepool="np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb" in resourcegroup="rg-candidate-5-0-v6nvmt4rxzjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a",
      "name": "203ad465-1e1f-4337-9659-f5fe2972a72a",
      "status": "Failed",
      "startTime": "2026-09-28T21:17:06.138034027Z",
      "endTime": "2026-09-28T21:36:06.809868093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m"
Unexpected error:
    <*fmt.wrapError | 0xc00087e120>: 
    failed to create NodePool np-rcls6m: failed waiting for nodepool="np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m" in resourcegroup="rg-candidate-4-21-92w77dwp5rg8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "name": "a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:31.628142216Z",
      "endTime": "2026-09-29T00:35:38.838060326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m"
Unexpected error:
    <*fmt.wrapError | 0xc00087e120>: 
    failed to create NodePool np-rcls6m: failed waiting for nodepool="np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m" in resourcegroup="rg-candidate-4-21-92w77dwp5rg8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "name": "a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:31.628142216Z",
      "endTime": "2026-09-29T00:35:38.838060326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc000b43f40>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc000b43f40>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb"
Unexpected error:
    <*fmt.wrapError | 0xc0010410e0>: 
    failed to create NodePool np-5m4wlb: failed waiting for nodepool="np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb" in resourcegroup="rg-candidate-4-23-wwwsqs8cghpq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052",
      "name": "91d9873c-e44e-4e44-ba41-ade87f500052",
      "status": "Failed",
      "startTime": "2026-09-29T00:18:19.018330415Z",
      "endTime": "2026-09-29T00:37:28.829514453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: Creating: virtualmachine creating or updating\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb"
Unexpected error:
    <*fmt.wrapError | 0xc0010410e0>: 
    failed to create NodePool np-5m4wlb: failed waiting for nodepool="np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb" in resourcegroup="rg-candidate-4-23-wwwsqs8cghpq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052",
      "name": "91d9873c-e44e-4e44-ba41-ade87f500052",
      "status": "Failed",
      "startTime": "2026-09-29T00:18:19.018330415Z",
      "endTime": "2026-09-29T00:37:28.829514453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: Creating: virtualmachine creating or updating\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000bad000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-64htmh" in resourcegroup="rg-np-noedge-64htmh-c4j9qdqfrq4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "name": "3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "status": "Failed",
      "startTime": "2026-09-29T01:00:25.564607872Z",
      "endTime": "2026-09-29T01:19:27.08336887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: Creating: virtualmachine creating or updating\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000bad000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-64htmh" in resourcegroup="rg-np-noedge-64htmh-c4j9qdqfrq4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "name": "3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "status": "Failed",
      "startTime": "2026-09-29T01:00:25.564607872Z",
      "endTime": "2026-09-29T01:19:27.08336887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: Creating: virtualmachine creating or updating\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-p28f7h" for cluster "cluster-candidate-4-22-p28f7h"
Unexpected error:
    <*fmt.wrapError | 0xc00068b3c0>: 
    failed to create NodePool np-p28f7h: failed waiting for nodepool="np-p28f7h" for cluster "cluster-candidate-4-22-p28f7h" in resourcegroup="rg-candidate-4-22-mrpj7hdjdh7l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/45638889-70a7-4fdc-8d99-520c3c98a19b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/45638889-70a7-4fdc-8d99-520c3c98a19b",
      "name": "45638889-70a7-4fdc-8d99-520c3c98a19b",
      "status": "Failed",
      "startTime": "2026-09-29T00:17:24.13464273Z",
      "endTime": "2026-09-29T00:36:28.832441888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a9f5u6l9p6p9x0x-np-p28f7h-pwbqm-k6srx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a9f5u6l9p6p9x0x-np-p28f7h-pwbqm-k6srx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-p28f7h" for cluster "cluster-candidate-4-22-p28f7h"
Unexpected error:
    <*fmt.wrapError | 0xc00068b3c0>: 
    failed to create NodePool np-p28f7h: failed waiting for nodepool="np-p28f7h" for cluster "cluster-candidate-4-22-p28f7h" in resourcegroup="rg-candidate-4-22-mrpj7hdjdh7l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/45638889-70a7-4fdc-8d99-520c3c98a19b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/45638889-70a7-4fdc-8d99-520c3c98a19b",
      "name": "45638889-70a7-4fdc-8d99-520c3c98a19b",
      "status": "Failed",
      "startTime": "2026-09-29T00:17:24.13464273Z",
      "endTime": "2026-09-29T00:36:28.832441888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a9f5u6l9p6p9x0x-np-p28f7h-pwbqm-k6srx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a9f5u6l9p6p9x0x-np-p28f7h-pwbqm-k6srx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007275a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdcdkl9rvn5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612",
      "name": "3a70e081-8150-479f-812b-31eb02522612",
      "status": "Failed",
      "startTime": "2026-09-29T01:01:02.974562787Z",
      "endTime": "2026-09-29T01:20:06.931508754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007275a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdcdkl9rvn5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612",
      "name": "3a70e081-8150-479f-812b-31eb02522612",
      "status": "Failed",
      "startTime": "2026-09-29T01:01:02.974562787Z",
      "endTime": "2026-09-29T01:20:06.931508754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b"
Unexpected error:
    <*fmt.wrapError | 0xc000b7f080>: 
    failed to create NodePool np-bhxv8b: failed waiting for nodepool="np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b" in resourcegroup="rg-candidate-4-21-mrwxnrjtcxks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "name": "b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:53.990233971Z",
      "endTime": "2026-09-29T00:35:58.987434024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b"
Unexpected error:
    <*fmt.wrapError | 0xc000b7f080>: 
    failed to create NodePool np-bhxv8b: failed waiting for nodepool="np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b" in resourcegroup="rg-candidate-4-21-mrwxnrjtcxks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "name": "b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:53.990233971Z",
      "endTime": "2026-09-29T00:35:58.987434024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f35488>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f35488>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00179cc60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-v4rx8l2tw9wq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00179cc60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-v4rx8l2tw9wq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000aa43c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qgqdfprzx89s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000aa43c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qgqdfprzx89s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001048440>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-k6t82b" resourcegroup="rg-np-dg-minor-k6t82b-4f2hjngd28k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-k6t82b-4f2hjngd28k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-k6t82b/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001048440>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-k6t82b" resourcegroup="rg-np-dg-minor-k6t82b-4f2hjngd28k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-k6t82b-4f2hjngd28k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-k6t82b/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9"
Unexpected error:
    <*fmt.wrapError | 0xc000d24340>: 
    failed to create NodePool np-5cw4w9: failed starting nodepool creation "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9" in resourcegroup="rg-nightly-4-20-b7gtl2svm55n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-b7gtl2svm55n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-5cw4w9/nodePools/np-5cw4w9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9"
Unexpected error:
    <*fmt.wrapError | 0xc000d24340>: 
    failed to create NodePool np-5cw4w9: failed starting nodepool creation "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9" in resourcegroup="rg-nightly-4-20-b7gtl2svm55n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-b7gtl2svm55n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-5cw4w9/nodePools/np-5cw4w9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0024d21b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-gd648czgpsjb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0024d21b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-gd648czgpsjb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc001055c80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tf5tm29lbrgl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "name": "51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "status": "Failed",
      "startTime": "2026-09-28T23:10:33.615095623Z",
      "endTime": "2026-09-28T23:29:40.114080242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2fhfr-24lbg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2fhfr-24lbg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc001055c80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tf5tm29lbrgl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "name": "51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "status": "Failed",
      "startTime": "2026-09-28T23:10:33.615095623Z",
      "endTime": "2026-09-28T23:29:40.114080242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2fhfr-24lbg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2fhfr-24lbg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000ade640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-t7jl46" in resourcegroup="rg-np-version-upgrade-t7jl46-65khsnvbc4zl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t7jl46-65khsnvbc4zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t7jl46/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000ade640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-t7jl46" in resourcegroup="rg-np-version-upgrade-t7jl46-65khsnvbc4zl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t7jl46-65khsnvbc4zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t7jl46/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc00130ef60>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-7g7qln9l2np5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "name": "ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "status": "Failed",
      "startTime": "2026-09-28T23:42:15.584467675Z",
      "endTime": "2026-09-29T00:01:22.906593334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc00130ef60>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-7g7qln9l2np5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "name": "ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "status": "Failed",
      "startTime": "2026-09-28T23:42:15.584467675Z",
      "endTime": "2026-09-29T00:01:22.906593334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0030a64a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kx45mg6gnp52" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b",
      "name": "7aa70862-e406-44df-915d-27100318e68b",
      "status": "Failed",
      "startTime": "2026-09-28T23:56:08.330966898Z",
      "endTime": "2026-09-29T00:15:12.920582798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0030a64a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kx45mg6gnp52" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b",
      "name": "7aa70862-e406-44df-915d-27100318e68b",
      "status": "Failed",
      "startTime": "2026-09-28T23:56:08.330966898Z",
      "endTime": "2026-09-29T00:15:12.920582798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000831500>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-zvgs4q" in resourcegroup="rg-np-skip-minor-zvgs4q-797p7px4gkts": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zvgs4q-797p7px4gkts/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zvgs4q/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000831500>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-zvgs4q" in resourcegroup="rg-np-skip-minor-zvgs4q-797p7px4gkts": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zvgs4q-797p7px4gkts/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zvgs4q/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-q9zwpnrqm4zb/cluster-nightly-5-1-sq64f4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110cda0>: 
    failed to create HCP cluster cluster-nightly-5-1-sq64f4: failed waiting for cluster="cluster-nightly-5-1-sq64f4" in resourcegroup="rg-nightly-5-1-q9zwpnrqm4zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec",
      "name": "25287b3f-b870-4e18-a51f-514432892aec",
      "status": "Failed",
      "startTime": "2026-09-28T23:31:45.913312Z",
      "endTime": "2026-09-29T00:00:52.759469958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-q9zwpnrqm4zb/cluster-nightly-5-1-sq64f4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110cda0>: 
    failed to create HCP cluster cluster-nightly-5-1-sq64f4: failed waiting for cluster="cluster-nightly-5-1-sq64f4" in resourcegroup="rg-nightly-5-1-q9zwpnrqm4zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec",
      "name": "25287b3f-b870-4e18-a51f-514432892aec",
      "status": "Failed",
      "startTime": "2026-09-28T23:31:45.913312Z",
      "endTime": "2026-09-29T00:00:52.759469958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a51a10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mnwb4zfspsdb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a51a10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mnwb4zfspsdb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0008a5e00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-282f9ccvqths/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0008a5e00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-282f9ccvqths/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l"
Unexpected error:
    <*fmt.wrapError | 0xc00171b4a0>: 
    failed to create NodePool np-nmsq2l: failed waiting for nodepool="np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l" in resourcegroup="rg-candidate-4-21-lf4lgs4pzp7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975",
      "name": "8303805f-b000-457e-b724-d4df7eac6975",
      "status": "Failed",
      "startTime": "2026-09-28T21:30:59.759776246Z",
      "endTime": "2026-09-28T21:50:00.145843425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l"
Unexpected error:
    <*fmt.wrapError | 0xc00171b4a0>: 
    failed to create NodePool np-nmsq2l: failed waiting for nodepool="np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l" in resourcegroup="rg-candidate-4-21-lf4lgs4pzp7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975",
      "name": "8303805f-b000-457e-b724-d4df7eac6975",
      "status": "Failed",
      "startTime": "2026-09-28T21:30:59.759776246Z",
      "endTime": "2026-09-28T21:50:00.145843425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001139d00>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001139d00>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8vn5sb" for cluster "cluster-candidate-4-20-8vn5sb"
Unexpected error:
    <*fmt.wrapError | 0xc00081f3e0>: 
    failed to create NodePool np-8vn5sb: failed waiting for nodepool="np-8vn5sb" for cluster "cluster-candidate-4-20-8vn5sb" in resourcegroup="rg-candidate-4-20-2kxj4bcd64pd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5b06b703-2ea9-4252-ab56-c1c6946ec407
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5b06b703-2ea9-4252-ab56-c1c6946ec407",
      "name": "5b06b703-2ea9-4252-ab56-c1c6946ec407",
      "status": "Failed",
      "startTime": "2026-09-28T21:21:03.360782437Z",
      "endTime": "2026-09-28T21:40:07.046700763Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-2kxj4bcd64pd-candidate-8vn5sb--managed/providers/Microsoft.Compute/virtualMachines/b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-2kxj4bcd64pd-candidate-8vn5sb--managed/providers/Microsoft.Compute/virtualMachines/b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8vn5sb" for cluster "cluster-candidate-4-20-8vn5sb"
Unexpected error:
    <*fmt.wrapError | 0xc00081f3e0>: 
    failed to create NodePool np-8vn5sb: failed waiting for nodepool="np-8vn5sb" for cluster "cluster-candidate-4-20-8vn5sb" in resourcegroup="rg-candidate-4-20-2kxj4bcd64pd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5b06b703-2ea9-4252-ab56-c1c6946ec407
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5b06b703-2ea9-4252-ab56-c1c6946ec407",
      "name": "5b06b703-2ea9-4252-ab56-c1c6946ec407",
      "status": "Failed",
      "startTime": "2026-09-28T21:21:03.360782437Z",
      "endTime": "2026-09-28T21:40:07.046700763Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-2kxj4bcd64pd-candidate-8vn5sb--managed/providers/Microsoft.Compute/virtualMachines/b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-2kxj4bcd64pd-candidate-8vn5sb--managed/providers/Microsoft.Compute/virtualMachines/b8s1d9p4q0u8w7k-np-8vn5sb-x9xrv-l4tr9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc0006d65e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tdccrn4dc9dh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1ed0a0-a90c-4672-a753-87b575007435
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1ed0a0-a90c-4672-a753-87b575007435",
      "name": "cf1ed0a0-a90c-4672-a753-87b575007435",
      "status": "Failed",
      "startTime": "2026-09-28T14:48:00.539019012Z",
      "endTime": "2026-09-28T15:07:01.911525796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-rhtj7-kpztn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tdccrn4dc9dh--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-rhtj7-kpztn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-rhtj7-kpztn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tdccrn4dc9dh--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-rhtj7-kpztn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc0006d65e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tdccrn4dc9dh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1ed0a0-a90c-4672-a753-87b575007435
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1ed0a0-a90c-4672-a753-87b575007435",
      "name": "cf1ed0a0-a90c-4672-a753-87b575007435",
      "status": "Failed",
      "startTime": "2026-09-28T14:48:00.539019012Z",
      "endTime": "2026-09-28T15:07:01.911525796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-rhtj7-kpztn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tdccrn4dc9dh--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-rhtj7-kpztn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-rhtj7-kpztn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tdccrn4dc9dh--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-rhtj7-kpztn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4z25qr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7lft27tlvnbf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0009fab20>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4z25qr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7lft27tlvnbf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0009fab20>: 
    storage account log verification failed
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d235a8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d235a8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001432880>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lllf2h" resourcegroup="rg-np-dg-minor-lllf2h-lmftzgmmkgfx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lllf2h-lmftzgmmkgfx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lllf2h/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001432880>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lllf2h" resourcegroup="rg-np-dg-minor-lllf2h-lmftzgmmkgfx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lllf2h-lmftzgmmkgfx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lllf2h/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000dbaaa0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-grgfk2" in resourcegroup="rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-grgfk2/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000dbaaa0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-grgfk2" in resourcegroup="rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-grgfk2/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdf28j9mzrjf/cluster-nightly-5-1-ds89w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a3e960>: 
    failed to create HCP cluster cluster-nightly-5-1-ds89w5: failed waiting for cluster="cluster-nightly-5-1-ds89w5" in resourcegroup="rg-nightly-5-1-vdf28j9mzrjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b",
      "name": "4cc98024-5165-40da-b846-ec5ee553964b",
      "status": "Failed",
      "startTime": "2026-09-29T00:13:16.920404751Z",
      "endTime": "2026-09-29T00:42:21.556617536Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdf28j9mzrjf/cluster-nightly-5-1-ds89w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a3e960>: 
    failed to create HCP cluster cluster-nightly-5-1-ds89w5: failed waiting for cluster="cluster-nightly-5-1-ds89w5" in resourcegroup="rg-nightly-5-1-vdf28j9mzrjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b",
      "name": "4cc98024-5165-40da-b846-ec5ee553964b",
      "status": "Failed",
      "startTime": "2026-09-29T00:13:16.920404751Z",
      "endTime": "2026-09-29T00:42:21.556617536Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a097a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-swk5ggttxwjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a097a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-swk5ggttxwjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00160fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-hfkhgg4cj7jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00160fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-hfkhgg4cj7jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0014c1ec0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k8xstznc2dd6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0014c1ec0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k8xstznc2dd6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001873458>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zwldz5957z9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "name": "00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "status": "Failed",
      "startTime": "2026-09-28T23:57:06.99965356Z",
      "endTime": "2026-09-29T00:16:13.233773533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-lwwnr-xksft: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-lwwnr-xksft: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001873458>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zwldz5957z9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "name": "00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "status": "Failed",
      "startTime": "2026-09-28T23:57:06.99965356Z",
      "endTime": "2026-09-29T00:16:13.233773533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-lwwnr-xksft: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-lwwnr-xksft: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000e91950>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-jnh726bgx45k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000e91950>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-jnh726bgx45k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000b459e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-sf9f9x57dtfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000b459e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-sf9f9x57dtfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8"
Unexpected error:
    <*fmt.wrapError | 0xc0012916a0>: 
    failed to create NodePool np-dpvlx8: failed waiting for nodepool="np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8" in resourcegroup="rg-nightly-5-0-5jrsqj92ljxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e",
      "name": "eee4d653-3b34-478d-9851-5151f132ff5e",
      "status": "Failed",
      "startTime": "2026-09-29T00:23:56.849718476Z",
      "endTime": "2026-09-29T00:43:01.619582837Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8"
Unexpected error:
    <*fmt.wrapError | 0xc0012916a0>: 
    failed to create NodePool np-dpvlx8: failed waiting for nodepool="np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8" in resourcegroup="rg-nightly-5-0-5jrsqj92ljxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e",
      "name": "eee4d653-3b34-478d-9851-5151f132ff5e",
      "status": "Failed",
      "startTime": "2026-09-29T00:23:56.849718476Z",
      "endTime": "2026-09-29T00:43:01.619582837Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x"
Unexpected error:
    <*fmt.wrapError | 0xc0008e6340>: 
    failed to create NodePool np-br2c4x: failed starting nodepool creation "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x" in resourcegroup="rg-nightly-4-20-bvth8x7dzstx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-bvth8x7dzstx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-br2c4x/nodePools/np-br2c4x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x"
Unexpected error:
    <*fmt.wrapError | 0xc0008e6340>: 
    failed to create NodePool np-br2c4x: failed starting nodepool creation "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x" in resourcegroup="rg-nightly-4-20-bvth8x7dzstx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-bvth8x7dzstx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-br2c4x/nodePools/np-br2c4x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc00088a3c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-7mm9jz" in resourcegroup="rg-np-skip-minor-7mm9jz-mkfnfjw9mlgl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-7mm9jz-mkfnfjw9mlgl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-7mm9jz/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc00088a3c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-7mm9jz" in resourcegroup="rg-np-skip-minor-7mm9jz-mkfnfjw9mlgl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-7mm9jz-mkfnfjw9mlgl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-7mm9jz/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001152660>: 
    resource group "rg-cp-ystream-upgrade-4-22-k576cn8fjplq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-c95x5l" in resourcegroup="rg-cp-ystream-upgrade-4-22-k576cn8fjplq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "name": "5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "status": "Failed",
      "startTime": "2026-09-28T22:54:48.090437049Z",
      "endTime": "2026-09-28T23:18:56.801422937Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t49njjvu97eroh1p7j9jssq535c6590 still exists (deletion dispatched at 2026-09-28T22:54:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001152660>: 
    resource group "rg-cp-ystream-upgrade-4-22-k576cn8fjplq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-c95x5l" in resourcegroup="rg-cp-ystream-upgrade-4-22-k576cn8fjplq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "name": "5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "status": "Failed",
      "startTime": "2026-09-28T22:54:48.090437049Z",
      "endTime": "2026-09-28T23:18:56.801422937Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t49njjvu97eroh1p7j9jssq535c6590 still exists (deletion dispatched at 2026-09-28T22:54:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred