time=2026-09-27T04:55:22.142Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2 description="Step kusto-grant-ingest\n  Kind: ARM\n  Template: templates/kusto-grant-ingest.bicep\n  Parameters: configurations/kusto-mgmt-ingest.tmpl.bicepparam"
time=2026-09-27T04:55:22.662Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2
time=2026-09-27T04:55:23.741Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2 deployment=2e905e8738711f49918794c12ee7bdb55d7d9202820d0f6abb23829107a4ce80 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8656896-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2e905e8738711f49918794c12ee7bdb55d7d9202820d0f6abb23829107a4ce80
time=2026-09-27T04:59:54.839Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kusto-grant-ingest stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8656896-mgmt-2/providers/Microsoft.Resources/deployments/2e905e8738711f49918794c12ee7bdb55d7d9202820d0f6abb23829107a4ce80/operationStatuses/08584111219623969246\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/grantKusto-hcp-ja2yqt7pclopi\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-kusto-us/providers/Microsoft.Kusto/clusters/hcp-dev-us-2/databases/HostedControlPlaneLogs/principalAssignments/grant-b6fd691c-fba0-50f9-bb53-13af017771ec\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Failed\\\",\\r\\n                \\\"message\\\": \\\"Internal Server Error\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:172]: failed to resolve test runner public IP
Unexpected error:
    <*fmt.wrapError | 0xc000858480>: 
    failed to query public IP echo service: Get "https://checkip.amazonaws.com": read tcp 172.24.8.136:46186->54.82.178.161:443: read: connection reset by peer
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:172]: failed to resolve test runner public IP
Unexpected error:
    <*fmt.wrapError | 0xc000858480>: 
    failed to query public IP echo service: Get "https://checkip.amazonaws.com": read tcp 172.24.8.136:46186->54.82.178.161:443: read: connection reset by peer
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T23:20:54Z
  Ended:       2026-09-27T23:44:54Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j9480704-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j9480704-mgmt-1
Namespace: ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:05:16Z
  Ended:       2026-09-27T22:39:15Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j2557696-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2557696-mgmt-2
Namespace: ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T20:30:25Z
  Ended:       2026-09-27T21:00:21Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster

Firing 2:
  State:       Resolved
  Started:     2026-09-27T20:30:25Z
  Ended:       2026-09-27T20:55:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T23:23:56Z
  Ended:       2026-09-28T00:09:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j9480704-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j9480704-mgmt-1
Namespace: ocm-arohcpci01-2t3l81mo181qkj3h3jcrvelvtdhmn5ea-a0v8t8r0i7d7k3n

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:08:16Z
  Ended:       2026-09-27T23:04:17Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j2557696-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2557696-mgmt-2
Namespace: ocm-arohcpci01-2t3k4ukpcu7g8mb4j4of7t2s9p7chht2-y0z1h5i8m3z9p7t

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T20:33:23Z
  Ended:       2026-09-27T21:25:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ipgo5l7305q17rs36aqaiigj71lve-agg-cluster

Firing 2:
  State:       Resolved
  Started:     2026-09-27T20:33:23Z
  Ended:       2026-09-27T21:20:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j0960384-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0960384-mgmt-1
Namespace: ocm-arohcpci01-2t3ip3j1bcpr5hi565ci3j9dn87t9gfa-g7x2p2x3z9l1o5m

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T18:00:15Z
  Ended:       2026-09-27T18:06:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1132032-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1132032-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T16:57:42Z
  Ended:       2026-09-27T17:30:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5652992-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5652992-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T08:29:34Z
  Ended:       2026-09-27T08:35:38Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0172672-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0172672-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:25:52Z
  Ended:       2026-09-27T19:47:50Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j1590528-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j1590528-mgmt-1
Namespace: ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T17:14:22Z
  Ended:       2026-09-27T17:26:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j5652992-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j5652992-mgmt-2
Namespace: ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T08:05:14Z
  Ended:       2026-09-27T08:31:16Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j0172672-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0172672-mgmt-1
Namespace: ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T19:28:50Z
  Ended:       2026-09-27T20:12:49Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j1590528-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j1590528-mgmt-1
Namespace: ocm-arohcpci01-2t3hqd26j6rv21sf05e07diagki1b9ln-f5s3p9a3j1f3z1q

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T17:21:23Z
  Ended:       2026-09-27T17:51:27Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j5652992-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j5652992-mgmt-2
Namespace: ocm-arohcpci01-2t3fn15tq7pskns61germc68tebdv165-z2y0q7d5f0a7u7e

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T08:08:19Z
  Ended:       2026-09-27T08:56:14Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j0172672-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0172672-mgmt-1
Namespace: ocm-arohcpci01-2t37ra42qcsvctongdu6ee8ia8ttj4b9-p6i6n4h8u0d8b6r

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000f6d380>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-4tsj8j" in resourcegroup="rg-np-noedge-4tsj8j-67tl4d7r7ztd" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "name": "36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "status": "Failed",
      "startTime": "2026-09-27T23:24:00.446121961Z",
      "endTime": "2026-09-27T23:43:03.337836309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000f6d380>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-4tsj8j" in resourcegroup="rg-np-noedge-4tsj8j-67tl4d7r7ztd" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "name": "36f4f936-b9bd-4488-b93c-580f6594d3e9",
      "status": "Failed",
      "startTime": "2026-09-27T23:24:00.446121961Z",
      "endTime": "2026-09-27T23:43:03.337836309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4tsj8j-67tl4d7r7ztd-np-ne-4tsj8j--managed/providers/Microsoft.Compute/virtualMachines/y7o9d9s7b3m0t4u-npnoedge-4-20-pwn57-gs8w5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00060efe0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wz5nqx5brl4f" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "name": "3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "status": "Failed",
      "startTime": "2026-09-27T23:20:03.595087025Z",
      "endTime": "2026-09-27T23:39:03.372293439Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-dq426: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-wz5nqx5brl4f--managed/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Network/networkInterfaces/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00060efe0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wz5nqx5brl4f" to finish creating: GET https://rp.j9480704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "name": "3e30fc72-f8f0-4109-846c-1ad5795f6f41",
      "status": "Failed",
      "startTime": "2026-09-27T23:20:03.595087025Z",
      "endTime": "2026-09-27T23:39:03.372293439Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-dq426: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-wz5nqx5brl4f--managed/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Network/networkInterfaces/u0j2v3g1y2f6r0v-np-1-6pwv6-dq426-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\nMachine u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wz5nqx5brl4f--managed/providers/Microsoft.Compute/virtualMachines/u0j2v3g1y2f6r0v-np-1-6pwv6-b4jpl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000fa8060>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-kcdxbmxql2zw" to finish creating: GET https://rp.j8939648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1",
      "name": "4c589deb-304d-4351-af73-1056e66b3ff1",
      "status": "Failed",
      "startTime": "2026-09-27T20:35:22.462990652Z",
      "endTime": "2026-09-27T20:54:31.807536747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000fa8060>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-kcdxbmxql2zw" to finish creating: GET https://rp.j8939648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c589deb-304d-4351-af73-1056e66b3ff1",
      "name": "4c589deb-304d-4351-af73-1056e66b3ff1",
      "status": "Failed",
      "startTime": "2026-09-27T20:35:22.462990652Z",
      "endTime": "2026-09-27T20:54:31.807536747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kcdxbmxql2zw--managed/providers/Microsoft.Compute/virtualMachines/b7y1q5d8i0e4m4p-autoscale-noaz-2mzsj-hlnq4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 13 time(s) (6 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:06:28Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T22:06:29Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-27T22:07:30Z
  Ended:       2026-09-27T22:48:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-27T22:07:30Z
  Ended:       2026-09-27T22:48:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-27T22:09:27Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-27T22:10:30Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd-0 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T17:37:18Z
  Ended:       2026-09-27T17:43:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1132032-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3g642vb3ovmhl4qiumvarv9qifaek4-c6e1u6h4n5k3a7j", pod="router-5584fc994b-5x9hp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3g642vb3ovmhl4qiumvarv9qifaek4-c6e1u6h4n5k3a7j/router-5584fc994b-5x9hp has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T14:40:20Z
  Ended:       2026-09-27T14:57:23Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5467008-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T01:21:42Z
  Ended:       2026-09-27T01:35:45Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8776064-svc", component="backend", name="datadump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T01:22:43Z
  Ended:       2026-09-27T01:37:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8776064-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000676bd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000676bd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002177290>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002177290>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000f972a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-t7rmzspt8s2k" to finish creating: GET https://rp.j6124032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "name": "0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "status": "Failed",
      "startTime": "2026-09-27T20:26:25.730010902Z",
      "endTime": "2026-09-27T20:45:26.626992422Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k: Failed: interfaces failed to create or update. err: failed to get existing resource nodepool-delete-t7rmzspt8s2k--managed/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-t7rmzspt8s2k--managed/providers/Microsoft.Network/networkInterfaces/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' seconds after additiona... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000f972a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-t7rmzspt8s2k" to finish creating: GET https://rp.j6124032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "name": "0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "status": "Failed",
      "startTime": "2026-09-27T20:26:25.730010902Z",
      "endTime": "2026-09-27T20:45:26.626992422Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k: Failed: interfaces failed to create or update. err: failed to get existing resource nodepool-delete-t7rmzspt8s2k--managed/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-t7rmzspt8s2k--managed/providers/Microsoft.Network/networkInterfaces/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' seconds after additiona... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00089fe00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-25nvltvpwf84" to finish creating: GET https://rp.j5652992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "name": "6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "status": "Failed",
      "startTime": "2026-09-27T17:01:59.070822701Z",
      "endTime": "2026-09-27T17:21:03.436757642Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-25l2v-tmrtf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-25l2v-tmrtf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00089fe00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-25nvltvpwf84" to finish creating: GET https://rp.j5652992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "name": "6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "status": "Failed",
      "startTime": "2026-09-27T17:01:59.070822701Z",
      "endTime": "2026-09-27T17:21:03.436757642Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-25l2v-tmrtf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-25l2v-tmrtf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de2a20>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-tcmg5wddbpw5" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "name": "75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:31.92972128Z",
      "endTime": "2026-09-27T22:26:36.766928511Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de2a20>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-tcmg5wddbpw5" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "name": "75ce2f03-ab4a-4df9-8666-ffa692e52e38",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:31.92972128Z",
      "endTime": "2026-09-27T22:26:36.766928511Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000f767a0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-tlzxr8jmz7kk" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "name": "4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.95420119Z",
      "endTime": "2026-09-27T22:26:17.083195861Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000f767a0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-tlzxr8jmz7kk" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "name": "4b1edad4-5029-44ea-8a67-f8504dd1dd95",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.95420119Z",
      "endTime": "2026-09-27T22:26:17.083195861Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f1e260>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hhxkjbgbq9fb" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "name": "3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.777648405Z",
      "endTime": "2026-09-27T22:26:17.06664737Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f1e260>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hhxkjbgbq9fb" to finish creating: GET https://rp.j0359424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "name": "3710ea21-b4a2-479d-839b-69b7c8bb9004",
      "status": "Failed",
      "startTime": "2026-09-27T21:57:15.777648405Z",
      "endTime": "2026-09-27T22:26:17.06664737Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 12 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-27T22:19:32Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-27T22:19:32Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-27T22:20:28Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-27T22:20:28Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd has not matched the expected number of replicas for longer than 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0029105a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-24vw7c2tw2cd" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-24vw7c2tw2cd, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0029105a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-24vw7c2tw2cd" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-24vw7c2tw2cd, error: context deadline exceeded
    ...
occurred

alert fired 8 time(s) (1 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T14:26:50Z
  Ended:       2026-09-27T14:32:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5453056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3d20g44qhuupsokqk6foneks2ohpk7-agg-cluster", pod="kube-apiserver-857c58cf4-hl9vw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t3d20g44qhuupsokqk6foneks2ohpk7-agg-cluster/kube-apiserver-857c58cf4-hl9vw has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc0007a2980>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-5fskdvpkxfwb" to finish creating: GET https://rp.j7860224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "name": "2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "status": "Failed",
      "startTime": "2026-09-27T05:48:02.556455592Z",
      "endTime": "2026-09-27T06:07:09.276038359Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2: Failed: interfaces failed to create or update. err: failed to get existing resource admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/providers/Microsoft.Network/networkInterfaces/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceeded. Please try again after '10' seconds after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc0007a2980>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-5fskdvpkxfwb" to finish creating: GET https://rp.j7860224.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "name": "2cbe44cd-36c6-4de6-8b35-e750b25ae85c",
      "status": "Failed",
      "startTime": "2026-09-27T05:48:02.556455592Z",
      "endTime": "2026-09-27T06:07:09.276038359Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2: Failed: interfaces failed to create or update. err: failed to get existing resource admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-5fskdvpkxfwb--managed/providers/Microsoft.Network/networkInterfaces/t6c5p7s4s6o4n0f-worker-jlhpf-j8xj2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceeded. Please try again after '10' seconds after... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:133]: failed to get admin REST config for cluster "cluster-candidate-4-21-cscwdf"
Unexpected error:
    <*fmt.wrapError | 0xc0001673c0>: 
    failed waiting for hcpCluster="cluster-candidate-4-21-cscwdf" in resourcegroup="rg-candidate-4-21-xbwd2trtds6r" to finish getting creds: GET https://rp.j6653824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:133]: failed to get admin REST config for cluster "cluster-candidate-4-21-cscwdf"
Unexpected error:
    <*fmt.wrapError | 0xc0001673c0>: 
    failed waiting for hcpCluster="cluster-candidate-4-21-cscwdf" in resourcegroup="rg-candidate-4-21-xbwd2trtds6r" to finish getting creds: GET https://rp.j6653824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75bf6e3d-ec48-419a-a251-e8975a2853cf"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T18:49:46Z
  Ended:       2026-09-27T18:56:46Z
  Severity:    Sev3
  Labels:      alertname="FleetControllerRetryHotLoop", cluster="ci01-j8175232-svc", component="fleet", name="nodepoolcontroller", region="westus3", severity="warning"
  Description: Fleet controller workqueue nodepoolcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.