alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:14:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j4475392-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/services-j4475392", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors"
  Description: More than 72% of completed cluster create (install) operations on ci01-j4475392-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T14:07:46Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j7129984-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7129984/providers/microsoft.monitor/accounts/services-j7129984", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors"
  Description: More than 72% of completed cluster create (install) operations on ci01-j7129984-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:38:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j4475392-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/services-j4475392", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j4475392-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T14:32:47Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j7129984-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7129984/providers/microsoft.monitor/accounts/services-j7129984", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j7129984-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc001389008>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.22.13) failed: no version in clusterversion/version status.history is greater than initial "4.22.13"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc001389008>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.22.13) failed: no version in clusterversion/version status.history is greater than initial "4.22.13"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c984b0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.33) failed: no version in clusterversion/version status.history is greater than initial "4.21.33"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c984b0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.33) failed: no version in clusterversion/version status.history is greater than initial "4.21.33"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:44]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc00011d9e0>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:44]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc00011d9e0>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:68]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc0014eed00>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:68]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc0014eed00>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.001s.
HCP resource requirements were not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with:
DataCurrent condition must be True
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.001s.
HCP resource requirements were not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with:
DataCurrent condition must be True
Expected
    <bool>: ...
to be true

time=2026-09-22T12:18:17.689Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T12:45:29.994Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T12:18:17.695906444Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-22T12:18:20.00208003Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T12:18:20.14445826Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T12:45:29.992055155Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"CF0FD6F1065F90B5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT8.5695733S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 44 for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T12:45:29.992133756Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DD6EAF8731EA851B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j2830976-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M23.5357859S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992147237Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9E2ABCCAB49D4C81\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j2830976-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M23.3548981S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992157287Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"42D7531EF6508071\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M53.0336442S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992166697Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1B516DD7916AE405\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:18:26.1875166Z\",\"duration\":\"PT26M47.0458862S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992178797Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2569996FCD48888B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2\",\"resourceName\":\"ci01-j2830976-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:18:26.1875166Z\",\"duration\":\"PT24M52.6018784S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992201978Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115273851533055\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 44 for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"27m12.296304024s\"}\n exit status 1"

time=2026-09-21T18:04:56.091Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T18:12:34.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T18:04:56.09869494Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-09-21T18:04:58.120073331Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T18:04:58.527982819Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-21T18:12:34.652011621Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"89CF033DB1DF73D0\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j7635328-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT1M56.3405526S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:12:34.652086242Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A0DB23EC9205187A\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j7635328-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT1M39.5985567S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:12:34.652107413Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"977918EE1130A569\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j7635328-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT4.3101609S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:12:34.652152984Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"5835781ECA9C15E0\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j7635328-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:05:03.7378518Z\",\"duration\":\"PT5M26.5601897S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:12:34.652204585Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E787274800217612\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j7635328-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT4.0601636S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:12:34.652246526Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B468A793AB498001\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2\",\"resourceName\":\"ci01-j7635328-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:05:03.7378518Z\",\"duration\":\"PT5M22.2477188S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:12:34.652279217Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115929865651702\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m38.553591047s\"}\n exit status 1"

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 15m0s timeout","severity":"error","time":"2026-09-23T00:52:30Z"}

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc001514030>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc001514030>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

time=2026-09-21T10:06:30.701Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T10:36:30.417Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T10:06:30.708972948Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4808064-mgmt-1\",\"clusterName\":\"ci01-j4808064-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-09-21T10:06:33.118843713Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4808064-mgmt-1\",\"clusterName\":\"ci01-j4808064-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T10:06:33.444184017Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4808064-mgmt-1\",\"clusterName\":\"ci01-j4808064-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

time=2026-09-22T11:23:53.670Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T11:53:53.299Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T11:23:53.695872086Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T11:23:55.867241647Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T11:23:55.995607642Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

time=2026-09-21T10:03:47.924Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T10:33:47.694Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T10:03:47.930202871Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8977280-mgmt-1\",\"clusterName\":\"ci01-j8977280-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-21T10:03:49.993282994Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8977280-mgmt-1\",\"clusterName\":\"ci01-j8977280-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T10:03:50.275505506Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8977280-mgmt-1\",\"clusterName\":\"ci01-j8977280-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

time=2026-09-22T07:33:19.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T08:03:18.587Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T07:33:19.023908785Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T07:33:21.03768985Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T07:33:21.167202045Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:293]: failed to create cluster role binding for external auth via VM
Unexpected error:
    <*errors.errorString | 0xc000aafea0>: 
    error: failed to create clusterrolebinding: etcdserver: request timed out, possibly due to previous leader failure
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:293]: failed to create cluster role binding for external auth via VM
Unexpected error:
    <*errors.errorString | 0xc000aafea0>: 
    error: failed to create clusterrolebinding: etcdserver: request timed out, possibly due to previous leader failure
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: Timed out after 623.703s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c664c0>: 
    failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token.
    Attempted credentials:
    	ClientSecretCredential: unable to resolve an endpoint: server response error:
    		 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: Timed out after 623.703s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c664c0>: 
    failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token.
    Attempted credentials:
    	ClientSecretCredential: unable to resolve an endpoint: server response error:
    		 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc00039e080>: 
    failed to load service config: failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc00039e080>: 
    failed to load service config: failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...

time=2026-09-22T13:40:24.327Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n"
time=2026-09-22T13:40:25.067Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpci01 kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:9e63919b9dde3a44483a565c55744abf9eab54314586f01a631a42b1104d5f2a] metricLabelsAllowlist:[nodes=[node.kubernetes.io/instance-type,kubernetes.azure.com/agentpool]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:be4e1b8bf4b8cc868cfa993818ed94e2536adc15713fc64942ce6a47d1bbabb9] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b840d65d98ba99f4d62329db721714b36ec23d7a0c8721d1fdb4fc839f97cf76]] thanosImage:map[registry:arohcpsvcdev.azurecr.io repository:thanos/thanos sha:b567818fe608067eb0f1d7c2c4fe361e7ad83c8a256234c97685f1d0bf670cc8 tag:v0.41.0] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[serviceAccount:map[managedIdentity:d7ccda51-6374-4c36-847f-8c68f35d13dc]] prometheusSpec:map[externalLabels:map[cluster:ci01-j6088704-svc environment:ci01 region:westus3] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:92fa8f5e91a0e02e0107c346f6c66657738dbf394946e52c208150fd498554bd] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-ci01-j6088704-svc-ybo2.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-7b09a45e774144ada99a9dc87a23c02f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 replicas:2 resources:map[limits:map[memory:NONE] requests:map[cpu:NONE memory:NONE]] shards:1 version: zoneCount:3]]"
time=2026-09-22T13:40:25.067Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-09-22T13:40:25.331Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus
time=2026-09-22T13:40:25.636Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false
time=2026-09-22T13:40:25.636Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-09-22T13:40:25.815Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-09-22T13:46:35.923Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-09-22T13:46:35.923Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
    <bool>: ...
to be false
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
    <bool>: ...
to be false

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hb6mtzv8hm6q" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81",
  "name": "b4e55441-95a5-466c-adac-c574d8bcdb81",
  "status": "Failed",
  "startTime": "2026-09-22T07:34:05.103384159Z",
  "endTime": "2026-09-22T07:53:06.353871266Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hb6mtzv8hm6q" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81",
  "name": "b4e55441-95a5-466c-adac-c574d8bcdb81",
  "status": "Failed",
  "startTime": "2026-09-22T07:34:05.103384159Z",
  "endTime": "2026-09-22T07:53:06.353871266Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-update-nodes: failed waiting for nodepool="np-update-nodes" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ckch4xg6nxh4" to finish creating: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84",
  "name": "e55eeddb-0ead-432e-a110-c696080abf84",
  "status": "Failed",
  "startTime": "2026-09-16T20:39:13.615216045Z",
  "endTime": "2026-09-16T20:58:16.977517435Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-update-nodes: failed waiting for nodepool="np-update-nodes" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ckch4xg6nxh4" to finish creating: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84",
  "name": "e55eeddb-0ead-432e-a110-c696080abf84",
  "status": "Failed",
  "startTime": "2026-09-16T20:39:13.615216045Z",
  "endTime": "2026-09-16T20:58:16.977517435Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b0090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mm2mhl4ksfsn" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66",
      "name": "5d1c688d-941e-4b98-9435-e64aad496c66",
      "status": "Failed",
      "startTime": "2026-09-23T00:01:29.977367864Z",
      "endTime": "2026-09-23T00:25:30.446435885Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c9ahk4qlhegp42spvj33akbiomek9 still exists (deletion dispatched at 2026-09-23T00:01:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b0090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mm2mhl4ksfsn" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66",
      "name": "5d1c688d-941e-4b98-9435-e64aad496c66",
      "status": "Failed",
      "startTime": "2026-09-23T00:01:29.977367864Z",
      "endTime": "2026-09-23T00:25:30.446435885Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c9ahk4qlhegp42spvj33akbiomek9 still exists (deletion dispatched at 2026-09-23T00:01:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015ac090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dbblg5cjpg9w" to finish deleting: GET https://rp.j9054592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "name": "0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "status": "Failed",
      "startTime": "2026-09-22T21:59:48.810888582Z",
      "endTime": "2026-09-22T22:23:48.20051154Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a3mthe182nict37k3988rs7tsgdm3 still exists (deletion dispatched at 2026-09-22T21:59:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015ac090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dbblg5cjpg9w" to finish deleting: GET https://rp.j9054592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "name": "0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "status": "Failed",
      "startTime": "2026-09-22T21:59:48.810888582Z",
      "endTime": "2026-09-22T22:23:48.20051154Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a3mthe182nict37k3988rs7tsgdm3 still exists (deletion dispatched at 2026-09-22T21:59:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00250e0a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-t2kdhs" in resourcegroup="rg-np-version-upgrade-t2kdhs-5vn4clcwghqp" to finish deleting: GET https://rp.j5645440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "name": "31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "status": "Failed",
      "startTime": "2026-09-22T21:50:57.166246559Z",
      "endTime": "2026-09-22T22:14:57.071797622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a2vdu59ka2it7qevdj75a5me0ng8e still exists (deletion dispatched at 2026-09-22T21:50:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00250e0a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-t2kdhs" in resourcegroup="rg-np-version-upgrade-t2kdhs-5vn4clcwghqp" to finish deleting: GET https://rp.j5645440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "name": "31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "status": "Failed",
      "startTime": "2026-09-22T21:50:57.166246559Z",
      "endTime": "2026-09-22T22:14:57.071797622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a2vdu59ka2it7qevdj75a5me0ng8e still exists (deletion dispatched at 2026-09-22T21:50:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:19:53Z
  Ended:       2026-09-23T00:25:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t0ca0g6h5bft9u4f3cil433bkiopk5i-t0j1r2j4w5n7o8z", pod="router-77497b854-mxvbx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t0ca0g6h5bft9u4f3cil433bkiopk5i-t0j1r2j4w5n7o8z/router-77497b854-mxvbx has been in a non-ready state for longer than 5 minutes.

alert fired 4 time(s) (1 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T13:15:15Z
  Ended:       2026-09-22T13:39:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8940032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t02sh3cjhqhk5sqr2356elqhegnu68u-v20261001", pod="router-6c76dd549c-ds2wz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t02sh3cjhqhk5sqr2356elqhegnu68u-v20261001/router-6c76dd549c-ds2wz has been in a non-ready state for longer than 5 minutes.

alert fired 23 time(s) (3 unknown, 20 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T04:06:07Z
  Ended:       2026-09-21T04:14:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9314560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv5p6n0mkeb9fggti8c47tsk9keuq7l-agg-cluster", pod="etcd-2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sv5p6n0mkeb9fggti8c47tsk9keuq7l-agg-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T04:07:06Z
  Ended:       2026-09-21T04:14:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9314560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster", pod="etcd-2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-21T04:07:06Z
  Ended:       2026-09-21T04:14:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9314560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster", pod="ignition-server-proxy-6ccf8c6444-24qfp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster/ignition-server-proxy-6ccf8c6444-24qfp has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000fb01e0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-22ntvrtddssz" to finish creating: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1",
      "name": "242f24a5-8f64-4654-9b39-e30267e94ed1",
      "status": "Failed",
      "startTime": "2026-09-22T13:01:52.830008497Z",
      "endTime": "2026-09-22T13:30:57.820386496Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 21m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000fb01e0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-22ntvrtddssz" to finish creating: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1",
      "name": "242f24a5-8f64-4654-9b39-e30267e94ed1",
      "status": "Failed",
      "startTime": "2026-09-22T13:01:52.830008497Z",
      "endTime": "2026-09-22T13:30:57.820386496Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 21m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000a06120>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wh7bztj94wh4" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "name": "a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:43.428610597Z",
      "endTime": "2026-09-19T21:03:49.814289269Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000a06120>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wh7bztj94wh4" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "name": "a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:43.428610597Z",
      "endTime": "2026-09-19T21:03:49.814289269Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d24520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fgvlnn5kdj25" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "name": "8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "status": "Failed",
      "startTime": "2026-09-19T20:35:08.292814934Z",
      "endTime": "2026-09-19T21:04:09.799274903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d24520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fgvlnn5kdj25" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "name": "8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "status": "Failed",
      "startTime": "2026-09-19T20:35:08.292814934Z",
      "endTime": "2026-09-19T21:04:09.799274903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T21:51:11Z
  Ended:       2026-09-22T21:57:11Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9054592-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9054592-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T21:41:14Z
  Ended:       2026-09-22T22:20:13Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5645440-svc", component="frontend", method="delete", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5645440-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T19:32:14Z
  Ended:       2026-09-22T19:38:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8706944-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8706944-svc).

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:00:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8t9zj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8t9zj (namespace arobit) on cluster ci01-j3118336-mgmt-1 has averaged 131.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:10:26Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-g6d4v", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-g6d4v (namespace arobit) on cluster ci01-j3118336-mgmt-2 has averaged 183.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:04:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b547f7d8b-bz92h", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b547f7d8b-bz92h (namespace mgmt-agent) on cluster ci01-j3118336-mgmt-1 has averaged 183.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:04:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-86c78bc956-rx5ct", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-86c78bc956-rx5ct (namespace clusters-service) on cluster ci01-j3118336-svc has averaged 829.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:08:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-vnkdq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-vnkdq (namespace arobit) on cluster ci01-j3118336-mgmt-1 has averaged 166.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:09:27Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b547f7d8b-pnvkw", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b547f7d8b-pnvkw (namespace mgmt-agent) on cluster ci01-j3118336-mgmt-2 has averaged 166.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T08:59:26Z
  Ended:       2026-09-17T09:17:25Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-86c78bc956-ljk5z", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-86c78bc956-ljk5z (namespace clusters-service) on cluster ci01-j3118336-svc has averaged 226.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:17:39Z
  Ended:       2026-09-17T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3085056-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-49zq5", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-49zq5 (namespace arobit) on cluster ci01-j3085056-svc has averaged 168.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T08:17:39Z
  Ended:       2026-09-17T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3085056-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-d4b6fd9f9-p4w2z", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-d4b6fd9f9-p4w2z (namespace aro-hcp) on cluster ci01-j3085056-svc has averaged 377.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T08:17:39Z
  Ended:       2026-09-17T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3085056-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-78587f7f-wtrr7", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-78587f7f-wtrr7 (namespace clusters-service) on cluster ci01-j3085056-svc has averaged 226.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T07:47:54Z
  Ended:       2026-09-17T07:53:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-648575ffd8-pgfs2", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-648575ffd8-pgfs2 (namespace clusters-service) on cluster ci01-j6077952-svc has averaged 276.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T07:52:53Z
  Ended:       2026-09-17T08:00:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-x8286", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-x8286 (namespace arobit) on cluster ci01-j6077952-mgmt-1 has averaged 128.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T07:54:53Z
  Ended:       2026-09-17T08:01:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-p74sm", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-p74sm (namespace arobit) on cluster ci01-j6077952-mgmt-1 has averaged 122.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-17T08:17:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-648575ffd8-526ht", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-648575ffd8-526ht (namespace clusters-service) on cluster ci01-j6077952-svc has averaged 185.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T00:53:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j7759104-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-pn2sd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-pn2sd (namespace arobit) on cluster ci01-j7759104-svc has averaged 127.3% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T22:23:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9327488-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-9ld5r", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-9ld5r (namespace arobit) on cluster ci01-j9327488-svc has averaged 126.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T20:26:03Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j0702464-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7j8m6", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-7j8m6 (namespace arobit) on cluster ci01-j0702464-svc has averaged 125.4% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89"
Unexpected error:
    <*fmt.wrapError | 0xc0011f1c80>: 
    failed to create NodePool np-gjkk89: failed waiting for nodepool="np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" in resourcegroup="rg-candidate-4-22-pm8qfxg2znfc" to finish creating: GET https://rp.j5377280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "name": "76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "status": "Failed",
      "startTime": "2026-09-22T22:15:39.485216693Z",
      "endTime": "2026-09-22T22:34:46.302814783Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89"
Unexpected error:
    <*fmt.wrapError | 0xc0011f1c80>: 
    failed to create NodePool np-gjkk89: failed waiting for nodepool="np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" in resourcegroup="rg-candidate-4-22-pm8qfxg2znfc" to finish creating: GET https://rp.j5377280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "name": "76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "status": "Failed",
      "startTime": "2026-09-22T22:15:39.485216693Z",
      "endTime": "2026-09-22T22:34:46.302814783Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x"
Unexpected error:
    <*fmt.wrapError | 0xc001027d20>: 
    failed to create NodePool np-55rf9x: failed waiting for nodepool="np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" in resourcegroup="rg-candidate-5-0-tv9lmfcf58jh" to finish creating: GET https://rp.j3849216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "name": "e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "status": "Failed",
      "startTime": "2026-09-22T15:06:40.249228102Z",
      "endTime": "2026-09-22T15:25:48.538295529Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x"
Unexpected error:
    <*fmt.wrapError | 0xc001027d20>: 
    failed to create NodePool np-55rf9x: failed waiting for nodepool="np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" in resourcegroup="rg-candidate-5-0-tv9lmfcf58jh" to finish creating: GET https://rp.j3849216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "name": "e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "status": "Failed",
      "startTime": "2026-09-22T15:06:40.249228102Z",
      "endTime": "2026-09-22T15:25:48.538295529Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b45e20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nvlgx5jfn6c7" to finish creating: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e",
      "name": "59212393-2ed7-490f-a765-56b81be64c8e",
      "status": "Failed",
      "startTime": "2026-09-22T12:57:41.103155101Z",
      "endTime": "2026-09-22T13:16:45.486929869Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b45e20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nvlgx5jfn6c7" to finish creating: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e",
      "name": "59212393-2ed7-490f-a765-56b81be64c8e",
      "status": "Failed",
      "startTime": "2026-09-22T12:57:41.103155101Z",
      "endTime": "2026-09-22T13:16:45.486929869Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:42:32Z
  Ended:       2026-09-23T00:53:32Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j6922368-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j6922368-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:20:40Z
  Ended:       2026-09-23T00:32:36Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j7723520-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j7723520-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T21:42:08Z
  Ended:       2026-09-22T22:16:09Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9054592-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9054592-svc).

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:20:53Z
  Ended:       2026-09-23T00:34:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T00:20:53Z
  Ended:       2026-09-23T00:39:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:43:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:43:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:45:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:45:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T10:39:33Z
  Ended:       2026-09-22T10:52:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-22T10:49:32Z
  Ended:       2026-09-22T10:56:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-bg4xww with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0016eec80>: 
    failed to create HCP cluster np-version-upgrade-cluster-bg4xww: failed waiting for cluster="np-version-upgrade-cluster-bg4xww" in resourcegroup="rg-np-version-upgrade-bg4xww-fxbpl7ffx59q" to finish creating: GET https://rp.j3962368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac",
      "name": "0f4e4436-53e4-4539-ab16-39eefac91dac",
      "status": "Failed",
      "startTime": "2026-09-22T23:45:41.203501418Z",
      "endTime": "2026-09-23T00:14:41.784000482Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-bg4xww with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0016eec80>: 
    failed to create HCP cluster np-version-upgrade-cluster-bg4xww: failed waiting for cluster="np-version-upgrade-cluster-bg4xww" in resourcegroup="rg-np-version-upgrade-bg4xww-fxbpl7ffx59q" to finish creating: GET https://rp.j3962368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac",
      "name": "0f4e4436-53e4-4539-ab16-39eefac91dac",
      "status": "Failed",
      "startTime": "2026-09-22T23:45:41.203501418Z",
      "endTime": "2026-09-23T00:14:41.784000482Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-gp68bk" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00011cba0>: 
    failed to create HCP cluster cluster-zstream-4-20-gp68bk: failed waiting for cluster="cluster-zstream-4-20-gp68bk" in resourcegroup="rg-zstream-upgrade-4-20-qcsqzkhckgg6" to finish creating: GET https://rp.j0115840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "name": "65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "status": "Failed",
      "startTime": "2026-09-22T16:56:56.107653498Z",
      "endTime": "2026-09-22T17:26:04.415591039Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-gp68bk" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00011cba0>: 
    failed to create HCP cluster cluster-zstream-4-20-gp68bk: failed waiting for cluster="cluster-zstream-4-20-gp68bk" in resourcegroup="rg-zstream-upgrade-4-20-qcsqzkhckgg6" to finish creating: GET https://rp.j0115840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "name": "65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "status": "Failed",
      "startTime": "2026-09-22T16:56:56.107653498Z",
      "endTime": "2026-09-22T17:26:04.415591039Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010ac9c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-smq89jgn4gq5" to finish creating: GET https://rp.j9233280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514",
      "name": "37dd9b79-8013-4222-b75a-587194eb2514",
      "status": "Failed",
      "startTime": "2026-09-22T10:03:21.992748868Z",
      "endTime": "2026-09-22T10:32:22.669673154Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010ac9c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-smq89jgn4gq5" to finish creating: GET https://rp.j9233280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514",
      "name": "37dd9b79-8013-4222-b75a-587194eb2514",
      "status": "Failed",
      "startTime": "2026-09-22T10:03:21.992748868Z",
      "endTime": "2026-09-22T10:32:22.669673154Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:05:55Z
  Ended:       2026-09-23T00:33:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T00:06:59Z
  Ended:       2026-09-23T00:37:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T17:17:06Z
  Ended:       2026-09-22T17:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-22T17:17:06Z
  Ended:       2026-09-22T17:49:03Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T10:24:38Z
  Ended:       2026-09-22T10:53:38Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-22T10:24:38Z
  Ended:       2026-09-22T10:50:38Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T16:52:14Z
  Ended:       2026-09-22T16:58:11Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3599104-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T15:47:19Z
  Ended:       2026-09-22T16:02:16Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5304576-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T14:26:28Z
  Ended:       2026-09-22T14:42:27Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5833600-svc", component="backend", name="datadump", region="westus3", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.