fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.006s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.006s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T00:54:17Z
  Ended:       2026-10-27T01:00:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1954816-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tmpuqe3u54c0c1kt3dhd472afk621tp-agg-cluster", pod="ignition-server-proxy-cccb6f6bd-9jw8t", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tmpuqe3u54c0c1kt3dhd472afk621tp-agg-cluster/ignition-server-proxy-cccb6f6bd-9jw8t has been in a non-ready state for longer than 5 minutes.

time=2026-10-26T23:19:14.439Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T23:19:15.908Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T23:19:17.082Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=facd493a02be1ad67e57b07e2fbd173b77c816ca0e9fe44bbdba2fb459fbeb85 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5967872-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ffacd493a02be1ad67e57b07e2fbd173b77c816ca0e9fe44bbdba2fb459fbeb85
time=2026-10-26T23:21:47.533Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.Resources/deployments/facd493a02be1ad67e57b07e2fbd173b77c816ca0e9fe44bbdba2fb459fbeb85/operationStatuses/08584085501288714898\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.Resources/deployments/cluster-wsenpawidiq6k\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j5967872-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j5967872-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-26T23:23:47.535Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T23:23:48.327Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T23:23:49.402Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=40b45e9bebcc2cf533d290687c03a069a8f711a71e9ba5f2836d47dbdaf9f362 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5967872-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F40b45e9bebcc2cf533d290687c03a069a8f711a71e9ba5f2836d47dbdaf9f362
time=2026-10-26T23:26:19.774Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5967872-svc/providers/Microsoft.Resources/deployments/40b45e9bebcc2cf533d290687c03a069a8f711a71e9ba5f2836d47dbdaf9f362/operationStatuses/08584085498565391842\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x37dc0b3be620>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        err: <*errors.errorString | 0x37dc0b40dbb0>{
            s: "dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x37dc0b3be620>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ffmm65nf5h4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        err: <*errors.errorString | 0x37dc0b40dbb0>{
            s: "dev/centralus/ci01-j5513088-mgmt-1/ocm-arohcpci01-2tmouldaggce1qoqe6i1onkn6rsmlu83-n4n8m4n5t5x8i6x router=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001: expected matching summary after test start and within 8m; summaryUID=03348f59-5de2-40fa-9a2e-5c7b8f5ae9bc node=aks-userswft3-23912472-vmss000001 pass=0c2770ec-0dde-47a1-b3d7-288aa5e5e851 at=2026-10-26T23:34:57Z",
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d"
Unexpected error:
    <*fmt.wrapError | 0x1a5d7d0bada0>: 
    failed to create NodePool np-csnz6d: failed waiting for nodepool="np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d" in resourcegroup="rg-candidate-5-0-x5kw7d69w7fz" to finish creating: GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6",
      "name": "27744955-3d23-4917-a3e3-fea7a42353f6",
      "status": "Failed",
      "startTime": "2026-10-26T22:49:05.815309572Z",
      "endTime": "2026-10-26T23:08:05.399599048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Compute/virtualMachines/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Network/networkInterfaces/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
            correlation ID: aca41f08-bc3a-4eb0-aeee-dbec04407cb6
            client request ID: 503f9ad7-c50a-4e8b-9147-306bbfbcb5a4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d"
Unexpected error:
    <*fmt.wrapError | 0x1a5d7d0bada0>: 
    failed to create NodePool np-csnz6d: failed waiting for nodepool="np-csnz6d" for cluster "cluster-candidate-5-0-csnz6d" in resourcegroup="rg-candidate-5-0-x5kw7d69w7fz" to finish creating: GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6",
      "name": "27744955-3d23-4917-a3e3-fea7a42353f6",
      "status": "Failed",
      "startTime": "2026-10-26T22:49:05.815309572Z",
      "endTime": "2026-10-26T23:08:05.399599048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Compute/virtualMachines/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-x5kw7d69w7fz-candidate-csnz6d--managed/providers/Microsoft.Network/networkInterfaces/l5s5h7x3f2f3n4c-np-csnz6d-9wh5b-gfx8x-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3418368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/27744955-3d23-4917-a3e3-fea7a42353f6
            correlation ID: aca41f08-bc3a-4eb0-aeee-dbec04407cb6
            client request ID: 503f9ad7-c50a-4e8b-9147-306bbfbcb5a4
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2148d2b87580>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x2148d2b87560>{
            observation: <*errors.errorString | 0x2148d2b63740>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x2148d2b87540>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2148d2b84b90>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2148d2b87500>{
                        msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2148d332b860>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2148d2b87580>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-plbw7phvf9sp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x2148d2b87560>{
            observation: <*errors.errorString | 0x2148d2b63740>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x2148d2b87540>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2148d2b84b90>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2148d2b87500>{
                        msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j9570944-mgmt-1\"; let ns = \"ocm-arohcpci01-2tmnl5tm8ds97r0ebee6bni26dkgbogo-q6u2e1q1r9e8l2c\"; let start = datetime(2026-10-26T21:38:32.1507932Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2148d332b860>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-23-99qq4z"
Unexpected error:
    <*errors.joinError | 0x39eb4f22870>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd": nodes "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x39eb302e180>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x39eb302e120>{
                    msg: "failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x39eb302e100>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x39eb3896140>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-23-99qq4z"
Unexpected error:
    <*errors.joinError | 0x39eb4f22870>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd": nodes "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x39eb302e180>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x39eb302e120>{
                    msg: "failed to label node x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x39eb302e100>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\": nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x39eb3896140>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "x7h3r3l6v2j3k1a-np-99qq4z-78ps4-4mpbd",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x75e04e9f9a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-lmxnc7" in resourcegroup="rg-np-version-upgrade-lmxnc7-q9mf48fq597k" to finish creating: GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "name": "d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "status": "Failed",
      "startTime": "2026-10-26T16:25:47.925761101Z",
      "endTime": "2026-10-26T16:44:52.193515957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
            correlation ID: f7b1fc07-103a-435d-9acb-75f17e0334fb
            client request ID: 9e0fce66-a04c-49ec-96a9-4c1b9377e626
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x75e04e9f9a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-lmxnc7" in resourcegroup="rg-np-version-upgrade-lmxnc7-q9mf48fq597k" to finish creating: GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "name": "d40573b8-7e51-417d-bd40-a08b0b1a57bd",
      "status": "Failed",
      "startTime": "2026-10-26T16:25:47.925761101Z",
      "endTime": "2026-10-26T16:44:52.193515957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
            correlation ID: f7b1fc07-103a-435d-9acb-75f17e0334fb
            client request ID: 9e0fce66-a04c-49ec-96a9-4c1b9377e626
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x3952e9095e40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x3952e9095e40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=13580, acm-agent=0",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xa75b49b000>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-c7rz5j8lqwzr" to finish creating: GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "name": "116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "status": "Failed",
      "startTime": "2026-10-26T16:04:30.855869275Z",
      "endTime": "2026-10-26T16:23:34.187561349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
            correlation ID: 0413d2ef-0809-4a88-a400-52222573fc14
            client request ID: 2d020f95-fbad-4853-8497-2a3822d5b948
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xa75b49b000>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-c7rz5j8lqwzr" to finish creating: GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "name": "116d7ad9-9cc2-4c9e-8347-2623016e8d01",
      "status": "Failed",
      "startTime": "2026-10-26T16:04:30.855869275Z",
      "endTime": "2026-10-26T16:23:34.187561349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
            correlation ID: 0413d2ef-0809-4a88-a400-52222573fc14
            client request ID: 2d020f95-fbad-4853-8497-2a3822d5b948
occurred

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:55:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.51:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-n8rf5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d66701f9-547c-4900-999a-9cc3801125e4"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-n8rf5 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:55:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.51:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-n8rf5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d66701f9-547c-4900-999a-9cc3801125e4"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-n8rf5 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:55:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r6g8v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d6ff7b03-cd37-4dcb-b713-5184927d807f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r6g8v (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-26T15:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8788992-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8788992/providers/microsoft.monitor/accounts/services-j8788992", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r6g8v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d6ff7b03-cd37-4dcb-b713-5184927d807f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r6g8v (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T16:08:39Z
  Ended:       2026-10-26T16:14:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8788992-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tmic1o03pdh91igeu5gq690e02l56kv-private-kas", pod="kube-scheduler-84685ff5d-cz6cm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tmic1o03pdh91igeu5gq690e02l56kv-private-kas/kube-scheduler-84685ff5d-cz6cm has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-rd6g78" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2215eed744c0>: 
    failed to create HCP cluster cluster-zstream-4-23-rd6g78: failed waiting for cluster="cluster-zstream-4-23-rd6g78" in resourcegroup="rg-zstream-upgrade-4-23-87rsp94v6tm5" to finish creating: GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "name": "aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "status": "Failed",
      "startTime": "2026-10-26T14:55:15.93882507Z",
      "endTime": "2026-10-26T15:24:15.95371054Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
            correlation ID: b0d46297-d5fe-44e3-bdff-447f2228b7af
            client request ID: fb0cf278-59cc-4ae8-8936-61b404d6c765
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-rd6g78" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2215eed744c0>: 
    failed to create HCP cluster cluster-zstream-4-23-rd6g78: failed waiting for cluster="cluster-zstream-4-23-rd6g78" in resourcegroup="rg-zstream-upgrade-4-23-87rsp94v6tm5" to finish creating: GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "name": "aa59838b-ffc9-4995-bb35-97d4c44beaff",
      "status": "Failed",
      "startTime": "2026-10-26T14:55:15.93882507Z",
      "endTime": "2026-10-26T15:24:15.95371054Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7915648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa59838b-ffc9-4995-bb35-97d4c44beaff
            correlation ID: b0d46297-d5fe-44e3-bdff-447f2228b7af
            client request ID: fb0cf278-59cc-4ae8-8936-61b404d6c765
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:16:58Z
  Ended:       2026-10-26T15:41:54Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T15:16:58Z
  Ended:       2026-10-26T15:41:55Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:18:52Z
  Ended:       2026-10-26T15:45:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9290d46d-5d55-4654-a54f-7d8738f6108c"
  Description: Pod ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-26T15:19:52Z
  Ended:       2026-10-26T15:45:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9290d46d-5d55-4654-a54f-7d8738f6108c"
  Description: Pod ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 8 time(s) (1 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:34:53Z
  Ended:       2026-10-26T15:40:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7915648-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tmhn1j616q7r7qo51i9d3pca09c16db-h2y9l6v1q7f6h4e", pod="cluster-network-operator-5f67bbdd9d-4fb4z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tmhn1j616q7r7qo51i9d3pca09c16db-h2y9l6v1q7f6h4e/cluster-network-operator-5f67bbdd9d-4fb4z has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T15:30:55Z
  Ended:       2026-10-26T15:41:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T15:30:55Z
  Ended:       2026-10-26T15:41:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7915648-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7915648/providers/microsoft.monitor/accounts/hcps-j7915648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-9766k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tmhnl5hj1n99osq72ci08bcfpe0967o-l3q7d3m3j2t0i9x/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x22cffa244a80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x22cffa418620>{
            s: "dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x22cffa244a80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ccfclw8d6p4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x22cffa418620>{
            s: "dev/centralus/ci01-j3696640-mgmt-2/ocm-arohcpci01-2tmgf7c99mhu459g7mk2ud5fkb7rv19p-h8m0e6o0u0k8h3e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

time=2026-10-26T12:51:59.398Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T12:52:00.891Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T12:52:02.109Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=255e1b4b8b6e4372d437a9506a320bc949b0cc72bb4ea9195aae5ed8cedae0ec portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6600576-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F255e1b4b8b6e4372d437a9506a320bc949b0cc72bb4ea9195aae5ed8cedae0ec
time=2026-10-26T12:55:02.788Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.Resources/deployments/255e1b4b8b6e4372d437a9506a320bc949b0cc72bb4ea9195aae5ed8cedae0ec/operationStatuses/08584085877640227854\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.Resources/deployments/cluster-pzvsdmxdkwyhw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j6600576-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j6600576-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-26T12:57:02.789Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-26T12:57:03.570Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-26T12:57:05.013Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4a6c02a4a9cb64920875392e566fb8079c7d7a854d89e39f64dab8204f190275 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6600576-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4a6c02a4a9cb64920875392e566fb8079c7d7a854d89e39f64dab8204f190275
time=2026-10-26T12:58:35.328Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6600576-svc/providers/Microsoft.Resources/deployments/4a6c02a4a9cb64920875392e566fb8079c7d7a854d89e39f64dab8204f190275/operationStatuses/08584085874612065441\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:122]: Timed out after 900.000s.
not all expected service metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:51 with:
expected maestro build info (MGMT) metrics but got no results (query: maestro_build_info)
Expected
    <[]prometheus.Result | len:0, cap:0>: []
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:122]: Timed out after 900.000s.
not all expected service metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/metrics_amw_svc.go:51 with:
expected maestro build info (MGMT) metrics but got no results (query: maestro_build_info)
Expected
    <[]prometheus.Result | len:0, cap:0>: []
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:131]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0x1b303bad2a00>: 
    failed to create service principal: create service principal for appId "a52f84ec-ecc1-43ca-9731-aac8f9f81ad5": Post "https://graph.microsoft.com/v1.0/servicePrincipals": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    {
        msg: "failed to create service principal: create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
        err: <*fmt.wrapError | 0x1b303bad2980>{
            msg: "create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
            err: <*url.Error | 0x1b303c6b42d0>{
                Op: "Post",
                URL: "https://graph.microsoft.com/v1.0/servicePrincipals",
                Err: <*http.timeoutError | 0x1b303d466f50>{
                    err: "net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:131]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0x1b303bad2a00>: 
    failed to create service principal: create service principal for appId "a52f84ec-ecc1-43ca-9731-aac8f9f81ad5": Post "https://graph.microsoft.com/v1.0/servicePrincipals": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    {
        msg: "failed to create service principal: create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
        err: <*fmt.wrapError | 0x1b303bad2980>{
            msg: "create service principal for appId \"a52f84ec-ecc1-43ca-9731-aac8f9f81ad5\": Post \"https://graph.microsoft.com/v1.0/servicePrincipals\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
            err: <*url.Error | 0x1b303c6b42d0>{
                Op: "Post",
                URL: "https://graph.microsoft.com/v1.0/servicePrincipals",
                Err: <*http.timeoutError | 0x1b303d466f50>{
                    err: "net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 603.481s.
Expected success, but got an error:
    <*errors.errorString | 0x12a5a347cf40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 603.481s.
Expected success, but got an error:
    <*errors.errorString | 0x12a5a347cf40>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8035, acm-agent=0",
    }

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:39:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-x7p5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6a5476df-33c8-40f9-ae9f-b4673c8f9f41"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-x7p5s (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:41:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.238:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r248f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="df5123e3-25f8-4237-b3ae-f6ee9c3dc14a"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r248f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:41:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.238:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-r248f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="df5123e3-25f8-4237-b3ae-f6ee9c3dc14a"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-r248f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-26T10:41:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2016512-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2016512/providers/microsoft.monitor/accounts/services-j2016512", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-x7p5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6a5476df-33c8-40f9-ae9f-b4673c8f9f41"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-x7p5s (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2d970cc94090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
      "name": "c8761660-38f6-4612-a201-a6ddec925f84",
      "status": "Failed",
      "startTime": "2026-10-26T09:38:13.907922394Z",
      "endTime": "2026-10-26T09:57:20.458727782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
            correlation ID: 4c993536-8323-4301-a24e-f6918a189628
            client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2d970cc94090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
      "name": "c8761660-38f6-4612-a201-a6ddec925f84",
      "status": "Failed",
      "startTime": "2026-10-26T09:38:13.907922394Z",
      "endTime": "2026-10-26T09:57:20.458727782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
            correlation ID: 4c993536-8323-4301-a24e-f6918a189628
            client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:78]: failed to create customer resources for ARM64 nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0x240d16aa66e0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-z9s25h" in resourcegroup="arm64-vm-cluster-r2854rq2nbt4" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ConflictError\",\r\n    \"message\": \"A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2147741\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
            correlation ID: 23c397c2-e55d-434d-a806-434e6889bce6
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:78]: failed to create customer resources for ARM64 nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0x240d16aa66e0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-z9s25h" in resourcegroup="arm64-vm-cluster-r2854rq2nbt4" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ConflictError\",\r\n    \"message\": \"A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2147741\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-r2854rq2nbt4/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-z9s25h/operationStatuses/08584086002951648222
            correlation ID: 23c397c2-e55d-434d-a806-434e6889bce6
            client request ID: unavailable
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xe6261437860>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xe6261437860>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11751, acm-agent=0",
    }

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:00:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdbj8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cf5b240e-dae8-4a44-acae-3a125b10e344"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdbj8 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:05:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-b6k74", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="259cbe08-9376-4204-b2ee-52499ae8c38f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-b6k74 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:05:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-b6k74", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="259cbe08-9376-4204-b2ee-52499ae8c38f"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-b6k74 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-26T08:05:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1073408-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1073408/providers/microsoft.monitor/accounts/services-j1073408", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdbj8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cf5b240e-dae8-4a44-acae-3a125b10e344"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdbj8 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d17e587640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zlz47zdxrs5r" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "name": "7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "status": "Failed",
      "startTime": "2026-10-26T07:28:38.842864831Z",
      "endTime": "2026-10-26T07:47:44.945885677Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
            correlation ID: 08c3cc6c-2bcb-4d71-83fb-d91ac0512763
            client request ID: dc9b7dea-fdfa-46b9-bb12-bc31e259fdc0
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1d17e587640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zlz47zdxrs5r" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "name": "7c622a94-9f78-4e75-b786-0aff8f0a726c",
      "status": "Failed",
      "startTime": "2026-10-26T07:28:38.842864831Z",
      "endTime": "2026-10-26T07:47:44.945885677Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7b4w8j4d4p7e3f-np-deletable-82q27-k447r: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c622a94-9f78-4e75-b786-0aff8f0a726c
            correlation ID: 08c3cc6c-2bcb-4d71-83fb-d91ac0512763
            client request ID: dc9b7dea-fdfa-46b9-bb12-bc31e259fdc0

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2ca8d1751180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5r2ss57dw6dn" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "name": "7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "status": "Failed",
      "startTime": "2026-10-26T07:10:55.24771548Z",
      "endTime": "2026-10-26T07:29:55.206194815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xtq4l-l6wv8: Failed: availabilitysets failed to create or update. err: failed to get existing resource ingress-cert-le-5r2ss57dw6dn--managed/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5r2ss57dw6dn--managed/providers/Microsoft.Compute/availabilitySets/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
            correlation ID: c6deec64-4edf-4f99-842e-fae9ee3bc0d5
            client request ID: 7c4b8c2b-3bc2-4ab4-98b5-cee20bd9e4da
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2ca8d1751180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5r2ss57dw6dn" to finish creating: GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "name": "7be36ffa-b81d-4c05-9801-1c0112e4103b",
      "status": "Failed",
      "startTime": "2026-10-26T07:10:55.24771548Z",
      "endTime": "2026-10-26T07:29:55.206194815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xtq4l-l6wv8: Failed: availabilitysets failed to create or update. err: failed to get existing resource ingress-cert-le-5r2ss57dw6dn--managed/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5r2ss57dw6dn--managed/providers/Microsoft.Compute/availabilitySets/2tmaoumf1ok3jc7bcp78t6cc6g7ou37b_ingress-cert-le-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '1' sec... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3286784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7be36ffa-b81d-4c05-9801-1c0112e4103b
            correlation ID: c6deec64-4edf-4f99-842e-fae9ee3bc0d5
            client request ID: 7c4b8c2b-3bc2-4ab4-98b5-cee20bd9e4da

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.003s.
Expected success, but got an error:
    <*errors.errorString | 0x294c3a4e6f50>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.003s.
Expected success, but got an error:
    <*errors.errorString | 0x294c3a4e6f50>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=9795, acm-agent=0",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xaf47cdc3b00>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-q2l2cwwbkpwv" to finish creating: GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb9e7887-b3a8-4bd3-8f18-28eec04446ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb9e7887-b3a8-4bd3-8f18-28eec04446ae",
      "name": "bb9e7887-b3a8-4bd3-8f18-28eec04446ae",
      "status": "Failed",
      "startTime": "2026-10-26T06:47:58.232968793Z",
      "endTime": "2026-10-26T07:07:05.709044225Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w7m0v1s2c9s1k0b-np-delete-main-2jlcl-5bwv9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w7m0v1s2c9s1k0b-np-delete-main-2jlcl-5bwv9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb9e7887-b3a8-4bd3-8f18-28eec04446ae
            correlation ID: d2413cb9-bbc4-41c5-8fb7-450d4d4c9295
            client request ID: 98336484-cdbd-4ed3-ad65-670b231d6a10
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xaf47cdc3b00>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-q2l2cwwbkpwv" to finish creating: GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb9e7887-b3a8-4bd3-8f18-28eec04446ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb9e7887-b3a8-4bd3-8f18-28eec04446ae",
      "name": "bb9e7887-b3a8-4bd3-8f18-28eec04446ae",
      "status": "Failed",
      "startTime": "2026-10-26T06:47:58.232968793Z",
      "endTime": "2026-10-26T07:07:05.709044225Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w7m0v1s2c9s1k0b-np-delete-main-2jlcl-5bwv9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w7m0v1s2c9s1k0b-np-delete-main-2jlcl-5bwv9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bb9e7887-b3a8-4bd3-8f18-28eec04446ae
            correlation ID: d2413cb9-bbc4-41c5-8fb7-450d4d4c9295
            client request ID: 98336484-cdbd-4ed3-ad65-670b231d6a10

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x3405dea41d00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-mjjkc7tsg6pw" to finish creating: GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "name": "152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "status": "Failed",
      "startTime": "2026-10-26T06:30:24.860566935Z",
      "endTime": "2026-10-26T06:49:25.947090385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
            correlation ID: 4ec64207-73fc-4c1b-8973-f8f47f8d6a7e
            client request ID: e0882a3c-3a9e-4bc0-8c16-c4da5403ce37
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x3405dea41d00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-mjjkc7tsg6pw" to finish creating: GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "name": "152da1d5-1d63-4b70-adbf-23b972dcdae5",
      "status": "Failed",
      "startTime": "2026-10-26T06:30:24.860566935Z",
      "endTime": "2026-10-26T06:49:25.947090385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2h8pd-mlvbj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-mjjkc7tsg6pw--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2h8pd-mlvbj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1234688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/152da1d5-1d63-4b70-adbf-23b972dcdae5
            correlation ID: 4ec64207-73fc-4c1b-8973-f8f47f8d6a7e
            client request ID: e0882a3c-3a9e-4bc0-8c16-c4da5403ce37

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-26T06:39:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1234688-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.55:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1234688/providers/microsoft.monitor/accounts/services-j1234688", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-79mdz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b0331707-593d-4709-aa13-977fd3c2bf42"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-79mdz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-26T06:39:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1234688-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.55:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1234688/providers/microsoft.monitor/accounts/services-j1234688", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-79mdz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b0331707-593d-4709-aa13-977fd3c2bf42"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-79mdz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-26T06:41:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1234688-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.65:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1234688/providers/microsoft.monitor/accounts/services-j1234688", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mjbv2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ceb7c145-a1d3-4aa0-b581-6b16c4403ca0"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mjbv2 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-26T06:42:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1234688-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.65:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1234688/providers/microsoft.monitor/accounts/services-j1234688", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mjbv2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ceb7c145-a1d3-4aa0-b581-6b16c4403ca0"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mjbv2 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:122]: cluster "cluster-hs-pre-546c5c" should complete its version rollout
Unexpected error:
    <*fmt.wrapError | 0x2e4bde720b40>: 
    VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39="Partial"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available
    {
        msg: "VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
        err: <*verifiers.pollObservationError | 0x2e4bde720ac0>{
            observation: <*errors.errorString | 0x2e4be03290f0>{
                s: "clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
            },
            interruption: <*fmt.wrapError | 0x2e4bde720aa0>{
                msg: "failed to get clusterversion \"version\": context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:122]: cluster "cluster-hs-pre-546c5c" should complete its version rollout
Unexpected error:
    <*fmt.wrapError | 0x2e4bde720b40>: 
    VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39="Partial"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available
    {
        msg: "VerifyClusterVersionRolloutCompleted timed out after 45m0s: clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
        err: <*verifiers.pollObservationError | 0x2e4bde720ac0>{
            observation: <*errors.errorString | 0x2e4be03290f0>{
                s: "clusterversion/version rollout not completed, expected every status.history entry to be Completed: history: 4.21.39=\"Partial\"; Failing=True: Cluster operator console is not available; Progressing=True: Unable to apply 4.21.39: the cluster operator console is not available",
            },
            interruption: <*fmt.wrapError | 0x2e4bde720aa0>{
                msg: "failed to get clusterversion \"version\": context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x3625ebf8a680>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gnkr42cnrl9g" to finish creating: GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549",
      "name": "c3982b99-da94-4694-b894-3515f5f00549",
      "status": "Failed",
      "startTime": "2026-10-26T06:24:09.477795034Z",
      "endTime": "2026-10-26T06:53:17.186342249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
            correlation ID: 9f2b05a4-0a2f-41bc-8784-edb61a547f4b
            client request ID: 3e8ce286-5388-4234-a7ff-6e068949d596
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x3625ebf8a680>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gnkr42cnrl9g" to finish creating: GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549",
      "name": "c3982b99-da94-4694-b894-3515f5f00549",
      "status": "Failed",
      "startTime": "2026-10-26T06:24:09.477795034Z",
      "endTime": "2026-10-26T06:53:17.186342249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3010560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3982b99-da94-4694-b894-3515f5f00549
            correlation ID: 9f2b05a4-0a2f-41bc-8784-edb61a547f4b
            client request ID: 3e8ce286-5388-4234-a7ff-6e068949d596
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T06:44:17Z
  Ended:       2026-10-26T07:09:15Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T06:45:17Z
  Ended:       2026-10-26T07:14:20Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T06:47:16Z
  Ended:       2026-10-26T07:14:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5b32bab-4c53-46e1-80dd-3f2bfd84e7a7"
  Description: Pod ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-26T06:47:16Z
  Ended:       2026-10-26T07:14:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5b32bab-4c53-46e1-80dd-3f2bfd84e7a7"
  Description: Pod ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T06:59:16Z
  Ended:       2026-10-26T07:10:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-26T06:59:17Z
  Ended:       2026-10-26T07:10:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3010560-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.218:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3010560/providers/microsoft.monitor/accounts/hcps-j3010560", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-jbwn4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tma82m9jgtb144qbrpk5j7np32jsq49-r5o4h3w2d4s3l7j/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x13f2532896c0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "status": "Failed",
      "startTime": "2026-10-26T03:58:11.830961438Z",
      "endTime": "2026-10-26T04:17:16.025648143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
            correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
            client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x13f2532896c0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
      "status": "Failed",
      "startTime": "2026-10-26T03:58:11.830961438Z",
      "endTime": "2026-10-26T04:17:16.025648143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
            correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
            client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1623311283c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x1623329ba440>{
            s: "dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1623311283c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-l578hzffxzwp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x1623329ba440>{
            s: "dev/westus3/ci01-j9769984-mgmt-2/ocm-arohcpci01-2tm7tusca27vplqd33uq4jnhru40s174-h2w4a5z0a2c9f5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x1cab81a88220>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-kfvwp2t79pvw" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403",
      "name": "b86345e2-8212-42f1-a199-1a2d8b540403",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:43.851254184Z",
      "endTime": "2026-10-26T04:05:45.382791214Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
            correlation ID: 63ad8909-3abd-481b-ad9d-99ee8f5ef24a
            client request ID: 2dbc76d4-ed2f-44e9-8bf3-81fac3665977
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x1cab81a88220>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-kfvwp2t79pvw" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403",
      "name": "b86345e2-8212-42f1-a199-1a2d8b540403",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:43.851254184Z",
      "endTime": "2026-10-26T04:05:45.382791214Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b86345e2-8212-42f1-a199-1a2d8b540403
            correlation ID: 63ad8909-3abd-481b-ad9d-99ee8f5ef24a
            client request ID: 2dbc76d4-ed2f-44e9-8bf3-81fac3665977
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x17d7d76a6b60>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n574rs4kgzmb" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "name": "0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:22.647294159Z",
      "endTime": "2026-10-26T04:05:25.39660378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, kube-scheduler, azure-cloud-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
            correlation ID: 2461ce1f-a2e3-4c70-ae70-58c7bc6e13e3
            client request ID: 87ad1faa-4a8b-48cb-888d-882999d849fb
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x17d7d76a6b60>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n574rs4kgzmb" to finish creating: GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "name": "0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e",
      "status": "Failed",
      "startTime": "2026-10-26T03:36:22.647294159Z",
      "endTime": "2026-10-26T04:05:25.39660378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, kube-scheduler, azure-cloud-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6195456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0a9279ed-f9e2-4b6a-ac1a-28f39ba0ae7e
            correlation ID: 2461ce1f-a2e3-4c70-ae70-58c7bc6e13e3
            client request ID: 87ad1faa-4a8b-48cb-888d-882999d849fb
occurred

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T03:51:15Z
  Ended:       2026-10-26T04:13:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g", pod="kube-scheduler-8557ff549-x2ncn", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g/kube-scheduler-8557ff549-x2ncn has been in a non-ready state for longer than 5 minutes.

alert fired 9 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-26T04:06:15Z
  Ended:       2026-10-26T04:15:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cloud-controller-manager", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g", pod="azure-cloud-controller-manager-8447d8b694-4zr5b", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d02104f0-060b-4edb-8a6c-e45f411809b0"
  Description: Pod ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g/azure-cloud-controller-manager-8447d8b694-4zr5b (cloud-controller-manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-26T04:06:16Z
  Ended:       2026-10-26T04:15:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cloud-controller-manager", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g", pod="azure-cloud-controller-manager-8447d8b694-4zr5b", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d02104f0-060b-4edb-8a6c-e45f411809b0"
  Description: Pod ocm-arohcpci01-2tm7pdravivuv2be1jlq3hjvbmiphefk-q7v3u6v3c8g7z4g/azure-cloud-controller-manager-8447d8b694-4zr5b (cloud-controller-manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:15:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7padnde7vvi5gn0v8i8h43umncmvl-cilium-cl", pod="csi-snapshot-controller-84dc6bbd7d-68jr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07c48f71-4da8-4382-849e-8180c459ebb1"
  Description: Pod ocm-arohcpci01-2tm7padnde7vvi5gn0v8i8h43umncmvl-cilium-cl/csi-snapshot-controller-84dc6bbd7d-68jr9 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="multus-admission-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="multus-admission-controller-6f7d476866-zcg5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8fa3adbb-8673-43f6-8f2c-cb4b4dc5a2ff"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/multus-admission-controller-6f7d476866-zcg5s (multus-admission-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:18:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cluster-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="cluster-policy-controller-7c486c777d-94qnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d0f7a448-44aa-43fc-891a-29f853013cc6"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/cluster-policy-controller-7c486c777d-94qnj (cluster-policy-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="multus-admission-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="multus-admission-controller-6f7d476866-zcg5s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8fa3adbb-8673-43f6-8f2c-cb4b4dc5a2ff"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/multus-admission-controller-6f7d476866-zcg5s (multus-admission-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Resolved
  Started:     2026-10-26T04:08:15Z
  Ended:       2026-10-26T04:18:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="cluster-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="cluster-policy-controller-7c486c777d-94qnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d0f7a448-44aa-43fc-891a-29f853013cc6"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/cluster-policy-controller-7c486c777d-94qnj (cluster-policy-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Resolved
  Started:     2026-10-26T04:09:15Z
  Ended:       2026-10-26T04:20:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="csi-snapshot-controller-56f74747dd-q2c9t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3dba828e-0851-4048-b70e-3efa4c0ac3c5"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/csi-snapshot-controller-56f74747dd-q2c9t (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 9:
  State:       Resolved
  Started:     2026-10-26T04:09:15Z
  Ended:       2026-10-26T04:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6195456-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6195456/providers/microsoft.monitor/accounts/hcps-j6195456", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics", pod="csi-snapshot-controller-56f74747dd-q2c9t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3dba828e-0851-4048-b70e-3efa4c0ac3c5"
  Description: Pod ocm-arohcpci01-2tm7pj2l0bb8pscrpmbg65ggbiohhe24-ksm-hcp-metrics/csi-snapshot-controller-56f74747dd-q2c9t (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").