CIHealth Failure Patterns

Environment: DEV

Failure patterns: 49 · Failures matched: 361 · Total runs: 118 · Runs affected: 99

Failure pattern

Failed at

Trend

Also in

alert [svc] userJourneyClusterProvisionErrors1h5m fired

full failure pattern:

alert [svc] userJourneyClusterProvisionErrors1h5m fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:14:23Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j4475392-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/services-j4475392", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors"
  Description: More than 72% of completed cluster create (install) operations on ci01-j4475392-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T14:07:46Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j7129984-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7129984/providers/microsoft.monitor/accounts/services-j7129984", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors"
  Description: More than 72% of completed cluster create (install) operations on ci01-j7129984-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors1h5m does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:46	PR #6889	2102424261724475392
2026-09-22 12:46	PR #6889	2102379019017129984

alert [svc] userJourneyClusterProvisionErrors6h30m fired

full failure pattern:

alert [svc] userJourneyClusterProvisionErrors6h30m fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:38:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j4475392-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/services-j4475392", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j4475392-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T14:32:47Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j7129984-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7129984/providers/microsoft.monitor/accounts/services-j7129984", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j7129984-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors6h30m does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:46	PR #6889	2102424261724475392
2026-09-22 12:46	PR #6889	2102379019017129984

DataCurrent condition must be True

full failure pattern:

DataCurrent condition must be True

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.001s.
HCP resource requirements were not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with:
DataCurrent condition must be True
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.001s.
HCP resource requirements were not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with:
DataCurrent condition must be True
Expected
    <bool>: ...
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Fleet should have HCP resource requirements data	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 12:46	PR #6889	2102379019017129984

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (1)

time=2026-09-22T12:18:17.689Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T12:45:29.994Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T12:18:17.695906444Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-22T12:18:20.00208003Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T12:18:20.14445826Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T12:45:29.992055155Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"CF0FD6F1065F90B5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT8.5695733S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 44 for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T12:45:29.992133756Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DD6EAF8731EA851B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j2830976-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M23.5357859S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992147237Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9E2ABCCAB49D4C81\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j2830976-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M23.3548981S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992157287Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"42D7531EF6508071\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M53.0336442S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992166697Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1B516DD7916AE405\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:18:26.1875166Z\",\"duration\":\"PT26M47.0458862S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992178797Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2569996FCD48888B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2\",\"resourceName\":\"ci01-j2830976-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:18:26.1875166Z\",\"duration\":\"PT24M52.6018784S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992201978Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115273851533055\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 44 for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"27m12.296304024s\"}\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 11:56	PR #6590	2102366391112830976

Process did not finish before 15m0s timeout

full failure pattern:

Process did not finish before 15m0s timeout

Signal:

unknown

none

Full failure examples (1)

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 15m0s timeout","severity":"error","time":"2026-09-23T00:52:30Z"}

Contributing tests (1)

Failed At	Job	Test	Support
unknown	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Job run should complete before timeout	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 22:46	PR #6359	2102530027982688256

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no s...

full failure pattern:

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc001514030>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc001514030>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 13:36	PR #7077	2102391598355582976

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j8724736-mgmt-2","clusterName":"ci01-j872473

Signal:

provision

none

Full failure examples (1)

time=2026-09-22T11:23:53.670Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T11:53:53.299Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T11:23:53.695872086Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T11:23:55.867241647Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T11:23:55.995607642Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 10:58	PR #7009	2102351739108724736

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j9153920-mgmt-2","clusterName":"ci01-j915392

Signal:

provision

none

Full failure examples (1)

time=2026-09-22T07:33:19.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T08:03:18.587Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T07:33:19.023908785Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T07:33:21.03768985Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T07:33:21.167202045Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 07:01	PR #6869	2102292246979153920

failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzure...

full failure pattern:

failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: Timed out after 623.703s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c664c0>: 
    failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token.
    Attempted credentials:
    	ClientSecretCredential: unable to resolve an endpoint: server response error:
    		 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: Timed out after 623.703s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c664c0>: 
    failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token.
    Attempted credentials:
    	ClientSecretCredential: unable to resolve an endpoint: server response error:
    		 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:00	PR #6756	2102412736506892288

failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1

full failure pattern:

failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1

Signal:

provision

none

Full failure examples (1)

time=2026-09-22T13:40:24.327Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n"
time=2026-09-22T13:40:25.067Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpci01 kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:9e63919b9dde3a44483a565c55744abf9eab54314586f01a631a42b1104d5f2a] metricLabelsAllowlist:[nodes=[node.kubernetes.io/instance-type,kubernetes.azure.com/agentpool]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:be4e1b8bf4b8cc868cfa993818ed94e2536adc15713fc64942ce6a47d1bbabb9] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b840d65d98ba99f4d62329db721714b36ec23d7a0c8721d1fdb4fc839f97cf76]] thanosImage:map[registry:arohcpsvcdev.azurecr.io repository:thanos/thanos sha:b567818fe608067eb0f1d7c2c4fe361e7ad83c8a256234c97685f1d0bf670cc8 tag:v0.41.0] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[serviceAccount:map[managedIdentity:d7ccda51-6374-4c36-847f-8c68f35d13dc]] prometheusSpec:map[externalLabels:map[cluster:ci01-j6088704-svc environment:ci01 region:westus3] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:92fa8f5e91a0e02e0107c346f6c66657738dbf394946e52c208150fd498554bd] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-ci01-j6088704-svc-ybo2.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-7b09a45e774144ada99a9dc87a23c02f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 replicas:2 resources:map[limits:map[memory:NONE] requests:map[cpu:NONE memory:NONE]] shards:1 version: zoneCount:3]]"
time=2026-09-22T13:40:25.067Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-09-22T13:40:25.331Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus
time=2026-09-22T13:40:25.636Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false
time=2026-09-22T13:40:25.636Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-09-22T13:40:25.815Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-09-22T13:46:35.923Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-09-22T13:46:35.923Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 13:01	PR #7066	2102382882726088704

scheduling requested CPU must be positive

full failure pattern:

scheduling requested CPU must be positive

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
    <bool>: ...
to be false
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
    <bool>: ...
to be false

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Fleet should have scheduling data for ready management clusters	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 12:46	PR #6889	2102379019017129984

--------------------------------------------------------------------------------

full failure pattern:

--------------------------------------------------------------------------------

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hb6mtzv8hm6q" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81",
  "name": "b4e55441-95a5-466c-adac-c574d8bcdb81",
  "status": "Failed",
  "startTime": "2026-09-22T07:34:05.103384159Z",
  "endTime": "2026-09-22T07:53:06.353871266Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hb6mtzv8hm6q" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81",
  "name": "b4e55441-95a5-466c-adac-c574d8bcdb81",
  "status": "Failed",
  "startTime": "2026-09-22T07:34:05.103384159Z",
  "endTime": "2026-09-22T07:53:06.353871266Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 06:37	PR #6995	2102286205365260288

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b0090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mm2mhl4ksfsn" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66",
      "name": "5d1c688d-941e-4b98-9435-e64aad496c66",
      "status": "Failed",
      "startTime": "2026-09-23T00:01:29.977367864Z",
      "endTime": "2026-09-23T00:25:30.446435885Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c9ahk4qlhegp42spvj33akbiomek9 still exists (deletion dispatched at 2026-09-23T00:01:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b0090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mm2mhl4ksfsn" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66",
      "name": "5d1c688d-941e-4b98-9435-e64aad496c66",
      "status": "Failed",
      "startTime": "2026-09-23T00:01:29.977367864Z",
      "endTime": "2026-09-23T00:25:30.446435885Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c9ahk4qlhegp42spvj33akbiomek9 still exists (deletion dispatched at 2026-09-23T00:01:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015ac090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dbblg5cjpg9w" to finish deleting: GET https://rp.j9054592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "name": "0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "status": "Failed",
      "startTime": "2026-09-22T21:59:48.810888582Z",
      "endTime": "2026-09-22T22:23:48.20051154Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a3mthe182nict37k3988rs7tsgdm3 still exists (deletion dispatched at 2026-09-22T21:59:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015ac090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dbblg5cjpg9w" to finish deleting: GET https://rp.j9054592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "name": "0b7ee6c5-b14b-4d17-a809-95e9f3fb536a",
      "status": "Failed",
      "startTime": "2026-09-22T21:59:48.810888582Z",
      "endTime": "2026-09-22T22:23:48.20051154Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a3mthe182nict37k3988rs7tsgdm3 still exists (deletion dispatched at 2026-09-22T21:59:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00250e0a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-t2kdhs" in resourcegroup="rg-np-version-upgrade-t2kdhs-5vn4clcwghqp" to finish deleting: GET https://rp.j5645440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "name": "31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "status": "Failed",
      "startTime": "2026-09-22T21:50:57.166246559Z",
      "endTime": "2026-09-22T22:14:57.071797622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a2vdu59ka2it7qevdj75a5me0ng8e still exists (deletion dispatched at 2026-09-22T21:50:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00250e0a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-t2kdhs" in resourcegroup="rg-np-version-upgrade-t2kdhs-5vn4clcwghqp" to finish deleting: GET https://rp.j5645440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "name": "31e9dfbb-4e6b-4928-875b-adfc122ac940",
      "status": "Failed",
      "startTime": "2026-09-22T21:50:57.166246559Z",
      "endTime": "2026-09-22T22:14:57.071797622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a2vdu59ka2it7qevdj75a5me0ng8e still exists (deletion dispatched at 2026-09-22T21:50:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (22)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	45
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	6
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	6
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1

Affected runs (65)

Date (UTC)	Associated PR	Prow job
2026-09-22 22:46	PR #6359	2102530027982688256
2026-09-22 20:20	PR #7084	2102495173459054592
2026-09-22 20:17	PR #6661	2102494708675645440
2026-09-22 20:00	PR #6821	2102494423198732288
2026-09-22 19:29	PR #6993	2102490037739327488
2026-09-22 19:15	PR #6925	2102479316322357248
2026-09-22 18:00	PR #5846	2102463947360702464
2026-09-22 17:48	PR #7020	2102458797258706944
2026-09-22 17:15	PR #4878	2102448371858935808
2026-09-22 17:01	PR #6821	2102443148557946880
2026-09-22 16:02	PR #7067	2102428355495202816
2026-09-22 15:50	PR #6661	2102425288674643968
2026-09-22 15:46	PR #6900	2102424385783599104
2026-09-22 15:37	PR #5846	2102421926960959488
2026-09-22 15:00	PR #6995	2102412746732605440
2026-09-22 15:00	PR #6756	2102412736506892288
2026-09-22 15:00	PR #6128	2102412694500937728
2026-09-22 14:44	PR #6364	2102408702790209536
2026-09-22 14:42	PR #7054	2102408218671058944
2026-09-22 14:38	PR #7020	2102407137983467520
2026-09-22 14:11	PR #6662	2102400420944023552
2026-09-22 14:08	PR #7072	2102399637812940800
2026-09-22 13:59	PR #6922	2102397479768035328
2026-09-22 13:22	PR #7001	2102388132350529536
2026-09-22 13:12	PR #6688	2102385490215833600
2026-09-22 13:02	PR #7067	2102382907266961408
2026-09-22 13:01	PR #6503	2102382852271247360
2026-09-22 12:15	PR #6364	2102371297248940032
2026-09-22 12:00	PR #6128	2102367426107674624
2026-09-22 12:00	PR #6995	2102367395485061120
2026-09-22 12:00	PR #6756	2102367380872105984
2026-09-22 11:29	PR #7070	2102359588845129728
2026-09-22 11:23	PR #6752	2102358117298737152
2026-09-22 11:04	PR #6092	2102353447767511040
2026-09-22 11:02	PR #6922	2102352929464782848
2026-09-22 10:11	PR #7067	2102340054952185856
2026-09-22 10:01	PR #7066	2102337463992193024
2026-09-22 10:01	PR #6503	2102337461471416320
2026-09-22 09:57	PR #6688	2102336356570107904
2026-09-22 09:02	PR #6756	2102322611546492928
2026-09-22 09:02	PR #6948	2102322606454607872
2026-09-22 09:02	PR #6995	2102322580831604736
2026-09-22 08:00	PR #6922	2102307266211352576
2026-09-22 07:14	PR #6503	2102295446469742592
2026-09-22 07:09	PR #6364	2102294129609281536
2026-09-22 06:46	PR #6128	2102288391285510144
2026-09-22 06:46	PR #7066	2102288344015704064
2026-09-22 06:37	PR #6995	2102286205365260288
2026-09-22 06:28	PR #7041	2102283819666116608
2026-09-22 06:00	PR #7053	2102276754965008384
2026-09-22 05:59	PR #5846	2102276703479926784
2026-09-22 05:59	PR #6948	2102276691408719872
2026-09-22 05:43	PR #6945	2102272452842229760
2026-09-22 03:59	PR #7060	2102246390443806720
2026-09-22 03:19	PR #6891	2102236197437837312
2026-09-22 03:00	PR #7053	2102231619598290944
2026-09-22 03:00	PR #5846	2102231550941728768
2026-09-22 03:00	PR #6945	2102231517219524608
2026-09-22 02:56	PR #5724	2102230542270337024
2026-09-22 02:55	PR #6948	2102230390138736640
2026-09-22 01:38	PR #7060	2102210924612947968
2026-09-22 01:10	PR #7061	2102203845328244736
2026-09-22 00:21	PR #7059	2102191495971868672
2026-09-22 00:02	PR #7053	2102186809273356288
2026-09-22 00:02	PR #6945	2102186737458483200

alert [svc] ServiceMemoryDrift fired

full failure pattern:

alert [svc] ServiceMemoryDrift fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T00:53:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j7759104-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-pn2sd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-pn2sd (namespace arobit) on cluster ci01-j7759104-svc has averaged 127.3% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T22:23:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9327488-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-9ld5r", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-9ld5r (namespace arobit) on cluster ci01-j9327488-svc has averaged 126.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T20:26:03Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j0702464-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7j8m6", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-7j8m6 (namespace arobit) on cluster ci01-j0702464-svc has averaged 125.4% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire	31

Affected runs (31)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #6821	2102534129747759104
2026-09-22 19:29	PR #6993	2102490037739327488
2026-09-22 18:00	PR #5846	2102463947360702464
2026-09-22 17:48	PR #7020	2102458797258706944
2026-09-22 17:15	PR #4878	2102448371858935808
2026-09-22 17:01	PR #6821	2102443148557946880
2026-09-22 15:46	PR #6900	2102424385783599104
2026-09-22 15:46	PR #6889	2102424261724475392
2026-09-22 15:37	PR #5846	2102421926960959488
2026-09-22 15:00	PR #6756	2102412736506892288
2026-09-22 15:00	PR #6128	2102412694500937728
2026-09-22 14:44	PR #6364	2102408702790209536
2026-09-22 14:08	PR #7072	2102399637812940800
2026-09-22 13:59	PR #6922	2102397479768035328
2026-09-22 13:22	PR #7001	2102388132350529536
2026-09-22 13:12	PR #6688	2102385490215833600
2026-09-22 12:00	PR #6948	2102367431941951488
2026-09-22 12:00	PR #6128	2102367426107674624
2026-09-22 11:29	PR #7070	2102359588845129728
2026-09-22 11:23	PR #6752	2102358117298737152
2026-09-22 10:01	PR #7066	2102337463992193024
2026-09-22 09:57	PR #6688	2102336356570107904
2026-09-22 09:08	PR #7041	2102324260725854208
2026-09-22 09:02	PR #6756	2102322611546492928
2026-09-22 09:02	PR #6995	2102322580831604736
2026-09-22 06:28	PR #7041	2102283819666116608
2026-09-22 03:19	PR #6891	2102236197437837312
2026-09-22 03:00	PR #5846	2102231550941728768
2026-09-22 01:38	PR #7060	2102210924612947968
2026-09-22 00:21	PR #7059	2102191495971868672
2026-09-22 00:02	PR #6945	2102186737458483200

alert [svc] FrontendPathMedianLatency fired

full failure pattern:

alert [svc] FrontendPathMedianLatency fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T21:51:11Z
  Ended:       2026-09-22T21:57:11Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9054592-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9054592-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T21:41:14Z
  Ended:       2026-09-22T22:20:13Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5645440-svc", component="frontend", method="delete", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5645440-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T19:32:14Z
  Ended:       2026-09-22T19:38:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8706944-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8706944-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire	21

Affected runs (21)

Date (UTC)	Associated PR	Prow job
2026-09-22 20:20	PR #7084	2102495173459054592
2026-09-22 20:17	PR #6661	2102494708675645440
2026-09-22 17:48	PR #7020	2102458797258706944
2026-09-22 17:15	PR #4878	2102448371858935808
2026-09-22 17:01	PR #6821	2102443148557946880
2026-09-22 15:00	PR #6995	2102412746732605440
2026-09-22 14:48	PR #6821	2102409636605857792
2026-09-22 14:38	PR #7020	2102407137983467520
2026-09-22 14:16	PR #7071	2102401634003849216
2026-09-22 13:22	PR #7001	2102388132350529536
2026-09-22 13:01	PR #6503	2102382852271247360
2026-09-22 12:00	PR #6128	2102367426107674624
2026-09-22 12:00	PR #6756	2102367380872105984
2026-09-22 10:01	PR #7066	2102337463992193024
2026-09-22 09:08	PR #7041	2102324260725854208
2026-09-22 09:02	PR #6948	2102322606454607872
2026-09-22 09:02	PR #6995	2102322580831604736
2026-09-22 08:00	PR #6922	2102307266211352576
2026-09-22 06:46	PR #7066	2102288344015704064
2026-09-22 05:59	PR #6948	2102276691408719872
2026-09-22 02:55	PR #6948	2102230390138736640

alert [svc] FrontendPathLatency fired

full failure pattern:

alert [svc] FrontendPathLatency fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:42:32Z
  Ended:       2026-09-23T00:53:32Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j6922368-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j6922368-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:20:40Z
  Ended:       2026-09-23T00:32:36Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j7723520-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j7723520-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T21:42:08Z
  Ended:       2026-09-22T22:16:09Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9054592-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9054592-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire	13

Affected runs (13)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #6922	2102534076756922368
2026-09-22 22:49	PR #7089	2102530666527723520
2026-09-22 20:20	PR #7084	2102495173459054592
2026-09-22 17:01	PR #6821	2102443148557946880
2026-09-22 12:15	PR #6364	2102371297248940032
2026-09-22 12:00	PR #6948	2102367431941951488
2026-09-22 10:01	PR #7066	2102337463992193024
2026-09-22 09:02	PR #6948	2102322606454607872
2026-09-22 08:58	PR #6516	2102321677739233280
2026-09-22 07:14	PR #6503	2102295446469742592
2026-09-22 05:59	PR #6948	2102276691408719872
2026-09-22 03:00	PR #5846	2102231550941728768
2026-09-22 00:02	PR #7053	2102186809273356288

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89"
Unexpected error:
    <*fmt.wrapError | 0xc0011f1c80>: 
    failed to create NodePool np-gjkk89: failed waiting for nodepool="np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" in resourcegroup="rg-candidate-4-22-pm8qfxg2znfc" to finish creating: GET https://rp.j5377280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "name": "76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "status": "Failed",
      "startTime": "2026-09-22T22:15:39.485216693Z",
      "endTime": "2026-09-22T22:34:46.302814783Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89"
Unexpected error:
    <*fmt.wrapError | 0xc0011f1c80>: 
    failed to create NodePool np-gjkk89: failed waiting for nodepool="np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" in resourcegroup="rg-candidate-4-22-pm8qfxg2znfc" to finish creating: GET https://rp.j5377280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "name": "76e0adbd-39c6-46a6-83d0-b884d2078d83",
      "status": "Failed",
      "startTime": "2026-09-22T22:15:39.485216693Z",
      "endTime": "2026-09-22T22:34:46.302814783Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x"
Unexpected error:
    <*fmt.wrapError | 0xc001027d20>: 
    failed to create NodePool np-55rf9x: failed waiting for nodepool="np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" in resourcegroup="rg-candidate-5-0-tv9lmfcf58jh" to finish creating: GET https://rp.j3849216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "name": "e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "status": "Failed",
      "startTime": "2026-09-22T15:06:40.249228102Z",
      "endTime": "2026-09-22T15:25:48.538295529Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x"
Unexpected error:
    <*fmt.wrapError | 0xc001027d20>: 
    failed to create NodePool np-55rf9x: failed waiting for nodepool="np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" in resourcegroup="rg-candidate-5-0-tv9lmfcf58jh" to finish creating: GET https://rp.j3849216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "name": "e6c15d22-5a6f-4305-86d0-76dd664e2eeb",
      "status": "Failed",
      "startTime": "2026-09-22T15:06:40.249228102Z",
      "endTime": "2026-09-22T15:25:48.538295529Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b45e20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nvlgx5jfn6c7" to finish creating: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e",
      "name": "59212393-2ed7-490f-a765-56b81be64c8e",
      "status": "Failed",
      "startTime": "2026-09-22T12:57:41.103155101Z",
      "endTime": "2026-09-22T13:16:45.486929869Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b45e20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nvlgx5jfn6c7" to finish creating: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e",
      "name": "59212393-2ed7-490f-a765-56b81be64c8e",
      "status": "Failed",
      "startTime": "2026-09-22T12:57:41.103155101Z",
      "endTime": "2026-09-22T13:16:45.486929869Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (8)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1

Affected runs (8)

Date (UTC)	Associated PR	Prow job
2026-09-22 21:12	PR #7053	2102506260145377280
2026-09-22 14:16	PR #7071	2102401634003849216
2026-09-22 12:00	PR #6756	2102367380872105984
2026-09-22 11:57	PR #6993	2102366747628670976
2026-09-22 09:02	PR #6128	2102322643775524864
2026-09-22 06:37	PR #6995	2102286205365260288
2026-09-22 06:28	PR #7041	2102283819666116608
2026-09-22 03:59	PR #7060	2102246390443806720

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0013740c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nzhgrm9s9j4m" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6",
      "name": "08842c66-3ce5-48d7-9b8b-2b725af2fdb6",
      "status": "Failed",
      "startTime": "2026-09-23T00:01:27.518475459Z",
      "endTime": "2026-09-23T00:25:30.539881421Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c8kgo0dundgseri9pe4b9l9rg3k7o still exists (deletion dispatched at 2026-09-23T00:01:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0013740c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nzhgrm9s9j4m" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6",
      "name": "08842c66-3ce5-48d7-9b8b-2b725af2fdb6",
      "status": "Failed",
      "startTime": "2026-09-23T00:01:27.518475459Z",
      "endTime": "2026-09-23T00:25:30.539881421Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c8kgo0dundgseri9pe4b9l9rg3k7o still exists (deletion dispatched at 2026-09-23T00:01:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00084a288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-68rjtkpsqm7b" to finish deleting: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d",
      "name": "6084bb0c-7df8-4842-ab2e-b53a4d94db1d",
      "status": "Failed",
      "startTime": "2026-09-22T19:42:12.055385484Z",
      "endTime": "2026-09-22T20:06:15.982381047Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t089ppdctio8jds461l0pe41vnps034 still exists (deletion dispatched at 2026-09-22T19:42:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00084a288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-68rjtkpsqm7b" to finish deleting: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d",
      "name": "6084bb0c-7df8-4842-ab2e-b53a4d94db1d",
      "status": "Failed",
      "startTime": "2026-09-22T19:42:12.055385484Z",
      "endTime": "2026-09-22T20:06:15.982381047Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t089ppdctio8jds461l0pe41vnps034 still exists (deletion dispatched at 2026-09-22T19:42:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0016de180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-8j45b7" in resourcegroup="rg-np-version-upgrade-8j45b7-xfd6p45fzpjm" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135",
      "name": "ef32d17b-db28-47e9-88f8-42656f04a135",
      "status": "Failed",
      "startTime": "2026-09-22T16:31:57.110038375Z",
      "endTime": "2026-09-22T16:56:06.840108207Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05ebc137s21nadehehjbelf4umr9sv still exists (deletion dispatched at 2026-09-22T16:31:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0016de180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-8j45b7" in resourcegroup="rg-np-version-upgrade-8j45b7-xfd6p45fzpjm" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135",
      "name": "ef32d17b-db28-47e9-88f8-42656f04a135",
      "status": "Failed",
      "startTime": "2026-09-22T16:31:57.110038375Z",
      "endTime": "2026-09-22T16:56:06.840108207Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05ebc137s21nadehehjbelf4umr9sv still exists (deletion dispatched at 2026-09-22T16:31:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (6)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-09-22 22:46	PR #6359	2102530027982688256
2026-09-22 18:00	PR #5846	2102463947360702464
2026-09-22 15:00	PR #6756	2102412736506892288
2026-09-22 12:00	PR #6948	2102367431941951488
2026-09-22 09:29	PR #7031	2102329484735680512
2026-09-22 03:00	PR #6945	2102231517219524608
2026-09-22 01:38	PR #7060	2102210924612947968

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T16:52:14Z
  Ended:       2026-09-22T16:58:11Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3599104-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T15:47:19Z
  Ended:       2026-09-22T16:02:16Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5304576-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T14:26:28Z
  Ended:       2026-09-22T14:42:27Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5833600-svc", component="backend", name="datadump", region="westus3", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:46	PR #6900	2102424385783599104
2026-09-22 14:46	PR #7079	2102409098145304576
2026-09-22 13:12	PR #6688	2102385490215833600
2026-09-22 10:01	PR #7066	2102337463992193024
2026-09-22 07:14	PR #6503	2102295446469742592
2026-09-22 03:59	PR #7060	2102246390443806720

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

full failure pattern:

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002455530>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002455530>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc0018dd510>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc0018dd510>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002204a80>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002204a80>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-22 18:00	PR #5846	2102463947360702464
2026-09-22 15:37	PR #5846	2102421926960959488
2026-09-22 14:44	PR #6364	2102408702790209536
2026-09-22 14:42	PR #7054	2102408218671058944

alert [svc] userJourneyFrontendLatency6h30m fired

full failure pattern:

alert [svc] userJourneyFrontendLatency6h30m fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T22:54:46Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j4395136-svc", component="slo", long_window="6h", region="westus3", severity="3", short_window="30m", slo="frontend-latency"
  Description: More than 6% of Frontend requests on cluster ci01-j4395136-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T22:28:47Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j9524736-svc", component="slo", long_window="6h", region="westus3", severity="3", short_window="30m", slo="frontend-latency"
  Description: More than 6% of Frontend requests on cluster ci01-j9524736-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T16:13:02Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j5857792-svc", component="slo", long_window="6h", region="canadacentral", severity="3", short_window="30m", slo="frontend-latency"
  Description: More than 6% of Frontend requests on cluster ci01-j5857792-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert userJourneyFrontendLatency6h30m does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-22 21:01	PR #5846	2102503575954395136
2026-09-22 20:46	PR #7064	2102499706289524736
2026-09-22 14:48	PR #6821	2102409636605857792
2026-09-22 10:01	PR #7066	2102337463992193024

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-bg4xww with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0016eec80>: 
    failed to create HCP cluster np-version-upgrade-cluster-bg4xww: failed waiting for cluster="np-version-upgrade-cluster-bg4xww" in resourcegroup="rg-np-version-upgrade-bg4xww-fxbpl7ffx59q" to finish creating: GET https://rp.j3962368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac",
      "name": "0f4e4436-53e4-4539-ab16-39eefac91dac",
      "status": "Failed",
      "startTime": "2026-09-22T23:45:41.203501418Z",
      "endTime": "2026-09-23T00:14:41.784000482Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-bg4xww with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0016eec80>: 
    failed to create HCP cluster np-version-upgrade-cluster-bg4xww: failed waiting for cluster="np-version-upgrade-cluster-bg4xww" in resourcegroup="rg-np-version-upgrade-bg4xww-fxbpl7ffx59q" to finish creating: GET https://rp.j3962368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac",
      "name": "0f4e4436-53e4-4539-ab16-39eefac91dac",
      "status": "Failed",
      "startTime": "2026-09-22T23:45:41.203501418Z",
      "endTime": "2026-09-23T00:14:41.784000482Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-gp68bk" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00011cba0>: 
    failed to create HCP cluster cluster-zstream-4-20-gp68bk: failed waiting for cluster="cluster-zstream-4-20-gp68bk" in resourcegroup="rg-zstream-upgrade-4-20-qcsqzkhckgg6" to finish creating: GET https://rp.j0115840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "name": "65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "status": "Failed",
      "startTime": "2026-09-22T16:56:56.107653498Z",
      "endTime": "2026-09-22T17:26:04.415591039Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-gp68bk" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00011cba0>: 
    failed to create HCP cluster cluster-zstream-4-20-gp68bk: failed waiting for cluster="cluster-zstream-4-20-gp68bk" in resourcegroup="rg-zstream-upgrade-4-20-qcsqzkhckgg6" to finish creating: GET https://rp.j0115840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "name": "65ff5f37-d01d-4885-92e1-a54bf276ecdf",
      "status": "Failed",
      "startTime": "2026-09-22T16:56:56.107653498Z",
      "endTime": "2026-09-22T17:26:04.415591039Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010ac9c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-smq89jgn4gq5" to finish creating: GET https://rp.j9233280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514",
      "name": "37dd9b79-8013-4222-b75a-587194eb2514",
      "status": "Failed",
      "startTime": "2026-09-22T10:03:21.992748868Z",
      "endTime": "2026-09-22T10:32:22.669673154Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010ac9c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-smq89jgn4gq5" to finish creating: GET https://rp.j9233280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514",
      "name": "37dd9b79-8013-4222-b75a-587194eb2514",
      "status": "Failed",
      "startTime": "2026-09-22T10:03:21.992748868Z",
      "endTime": "2026-09-22T10:32:22.669673154Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #7064	2102534053163962368
2026-09-22 16:02	PR #6503	2102428363070115840
2026-09-22 08:58	PR #6516	2102321677739233280

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:05:55Z
  Ended:       2026-09-23T00:33:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T00:06:59Z
  Ended:       2026-09-23T00:37:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T17:17:06Z
  Ended:       2026-09-22T17:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-22T17:17:06Z
  Ended:       2026-09-22T17:49:03Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T10:24:38Z
  Ended:       2026-09-22T10:53:38Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-22T10:24:38Z
  Ended:       2026-09-22T10:50:38Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #7064	2102534053163962368
2026-09-22 16:02	PR #6503	2102428363070115840
2026-09-22 08:58	PR #6516	2102321677739233280

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:20:53Z
  Ended:       2026-09-23T00:34:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T00:20:53Z
  Ended:       2026-09-23T00:39:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:43:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:43:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:45:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-22T17:45:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T10:39:33Z
  Ended:       2026-09-22T10:52:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-22T10:49:32Z
  Ended:       2026-09-22T10:56:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #7064	2102534053163962368
2026-09-22 15:46	PR #6889	2102424261724475392
2026-09-22 08:58	PR #6516	2102321677739233280

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T23:53:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-5x4b6", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-5x4b6 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T23:53:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-56wq6", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-56wq6 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-22T23:53:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-njbgt", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-njbgt has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-22T23:53:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-dkjc6", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-dkjc6 has been in a non-ready state for longer than 5 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T21:56:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-2hfrg", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-2hfrg has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T21:56:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-xc5hd", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-xc5hd has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-22T21:56:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-74xjw", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-74xjw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-22T21:56:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-cnj5s", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-cnj5s has been in a non-ready state for longer than 5 minutes.

alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T19:55:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-8sx8x", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-8sx8x has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T19:55:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-zpdt2", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-zpdt2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-22T19:55:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-pc755", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-pc755 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-22T19:55:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-kzj4q", region="centralus", severity="warning"
  Description: Pod arobit/arobit-forwarder-kzj4q has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-22T19:57:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3875072-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-749558849-6khd4", region="centralus", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-749558849-6khd4 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #6820	2102534072453566464
2026-09-22 21:01	PR #6820	2102503516554661888
2026-09-22 18:22	PR #6820	2102471925983875072

alert [svc] ArobitForwarderJobUp fired

full failure pattern:

alert [svc] ArobitForwarderJobUp fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T00:04:14Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j3566464-svc", component="monitoring-infrastructure", region="centralus", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j3566464-svc has been unreachable for 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T22:06:23Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j4661888-svc", component="monitoring-infrastructure", region="westus3", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j4661888-svc has been unreachable for 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T20:05:27Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j3875072-svc", component="monitoring-infrastructure", region="centralus", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j3875072-svc has been unreachable for 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ArobitForwarderJobUp does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #6820	2102534072453566464
2026-09-22 21:01	PR #6820	2102503516554661888
2026-09-22 18:22	PR #6820	2102471925983875072

alert [svc] KubeDaemonSetRolloutStuck fired

full failure pattern:

alert [svc] KubeDaemonSetRolloutStuck fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T00:03:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.65:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3566464/providers/microsoft.monitor/accounts/services-j3566464", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lcpcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T00:03:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.65:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3566464/providers/microsoft.monitor/accounts/services-j3566464", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lcpcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T22:06:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4661888/providers/microsoft.monitor/accounts/services-j4661888", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lrr2t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T22:06:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4661888/providers/microsoft.monitor/accounts/services-j4661888", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lrr2t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T20:05:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.70:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3875072/providers/microsoft.monitor/accounts/services-j3875072", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lbmz5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T20:05:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.70:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3875072/providers/microsoft.monitor/accounts/services-j3875072", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lbmz5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #6820	2102534072453566464
2026-09-22 21:01	PR #6820	2102503516554661888
2026-09-22 18:22	PR #6820	2102471925983875072

failed to roll out Helm release: context deadline exceeded

full failure pattern:

failed to roll out Helm release: context deadline exceeded

Signal:

provision

none

Full failure examples (3)

time=2026-09-22T23:47:19.977Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T23:47:20.376Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3566464-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3566464-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:3739bfae-90c4-46a6-9e9a-788bc6da855b region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#3739bfae-90c4-46a6-9e9a-788bc6da855b INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T23:47:20.376Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:47:20.465Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T23:47:20.652Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T23:47:20.652Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:47:20.723Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:57:23.239Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T23:57:23.240Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T23:58:23.243Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T23:58:23.878Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3566464-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3566464-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:3739bfae-90c4-46a6-9e9a-788bc6da855b region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#3739bfae-90c4-46a6-9e9a-788bc6da855b INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T23:58:23.878Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:58:24.112Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T23:58:24.439Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:58:24.439Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:58:24.639Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-22T23:58:25.408Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:58:25.408Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-22T23:58:25.669Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-22T23:58:26.134Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-22T23:58:26.496Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-22T23:58:26.679Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-22T23:58:26.994Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-22T23:58:27.236Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-22T23:58:27.460Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T23:58:27.460Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T23:58:27.560Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-23T00:08:29.375Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-23T00:08:29.375Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-23T00:09:29.384Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-23T00:09:29.909Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3566464-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3566464-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:3739bfae-90c4-46a6-9e9a-788bc6da855b region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#3739bfae-90c4-46a6-9e9a-788bc6da855b INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-23T00:09:29.909Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T00:09:30.097Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-23T00:09:30.328Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T00:09:30.328Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T00:09:30.580Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-23T00:09:31.341Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T00:09:31.341Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-23T00:09:31.535Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-23T00:09:31.879Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-23T00:09:32.169Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-23T00:09:32.570Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-23T00:09:32.841Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-23T00:09:33.156Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-23T00:09:33.376Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-23T00:09:33.376Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T00:09:33.452Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-23T00:17:19.905Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-23T00:17:19.906Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

time=2026-09-22T21:50:03.851Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T21:50:04.426Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j4661888-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j4661888-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:8df94b5e-187a-4556-af6a-9b474ecc0b97 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#8df94b5e-187a-4556-af6a-9b474ecc0b97 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T21:50:04.426Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T21:50:04.653Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T21:50:04.887Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T21:50:04.887Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T21:50:05.009Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:00:08.502Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T22:00:08.503Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T22:01:08.503Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T22:01:09.036Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j4661888-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j4661888-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:8df94b5e-187a-4556-af6a-9b474ecc0b97 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#8df94b5e-187a-4556-af6a-9b474ecc0b97 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T22:01:09.036Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:01:09.387Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T22:01:09.975Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:01:09.975Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:01:10.243Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-22T22:01:11.277Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:01:11.277Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:01:11.584Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:01:12.230Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-22T22:01:13.220Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-22T22:01:13.572Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:01:14.205Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:01:14.723Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:01:15.282Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T22:01:15.282Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:01:15.520Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-22T22:11:18.108Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T22:11:18.108Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T22:12:18.116Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T22:12:19.145Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j4661888-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j4661888-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:8df94b5e-187a-4556-af6a-9b474ecc0b97 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#8df94b5e-187a-4556-af6a-9b474ecc0b97 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T22:12:19.145Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:12:19.438Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T22:12:19.970Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:12:19.970Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:12:20.187Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-22T22:12:21.247Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:12:21.247Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:12:21.585Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:12:22.170Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-22T22:12:22.930Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-22T22:12:23.284Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:12:23.764Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:12:24.305Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-22T22:12:24.890Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T22:12:24.890Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T22:12:25.135Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-22T22:20:03.973Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-22T22:20:03.973Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

time=2026-09-22T19:48:43.126Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T19:48:43.619Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3875072-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3875072-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:f4808719-7718-4dd5-a60e-1d3b526f1087 region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#f4808719-7718-4dd5-a60e-1d3b526f1087 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T19:48:43.619Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:48:43.778Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T19:48:43.960Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T19:48:43.960Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:48:44.045Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:58:46.492Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T19:58:46.492Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T19:59:46.495Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T19:59:47.378Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3875072-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3875072-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:f4808719-7718-4dd5-a60e-1d3b526f1087 region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#f4808719-7718-4dd5-a60e-1d3b526f1087 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T19:59:47.379Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:59:47.630Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T19:59:48.103Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:59:48.103Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:59:48.332Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-22T19:59:49.351Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:59:49.352Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-22T19:59:49.594Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-22T19:59:49.976Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-22T19:59:50.450Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-22T19:59:50.618Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-22T19:59:50.774Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-22T19:59:51.169Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-22T19:59:51.518Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T19:59:51.518Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T19:59:51.622Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-22T20:09:53.731Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T20:09:53.731Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-22T20:10:53.735Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-22T20:10:54.409Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3875072-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3875072-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:f4808719-7718-4dd5-a60e-1d3b526f1087 region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#f4808719-7718-4dd5-a60e-1d3b526f1087 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-22T20:10:54.409Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T20:10:54.723Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-22T20:10:55.205Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T20:10:55.205Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T20:10:55.345Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-22T20:10:56.191Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T20:10:56.191Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-22T20:10:56.565Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-22T20:10:56.839Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-22T20:10:57.169Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-22T20:10:57.365Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-22T20:10:57.525Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-22T20:10:57.721Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-22T20:10:58.033Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-22T20:10:58.033Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-22T20:10:58.139Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-22T20:18:43.221Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-22T20:18:43.222Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/arobit	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #6820	2102534072453566464
2026-09-22 21:01	PR #6820	2102503516554661888
2026-09-22 18:22	PR #6820	2102471925983875072

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:19:53Z
  Ended:       2026-09-23T00:25:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t0ca0g6h5bft9u4f3cil433bkiopk5i-t0j1r2j4w5n7o8z", pod="router-77497b854-mxvbx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t0ca0g6h5bft9u4f3cil433bkiopk5i-t0j1r2j4w5n7o8z/router-77497b854-mxvbx has been in a non-ready state for longer than 5 minutes.

alert fired 4 time(s) (1 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T13:15:15Z
  Ended:       2026-09-22T13:39:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8940032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t02sh3cjhqhk5sqr2356elqhegnu68u-v20261001", pod="router-6c76dd549c-ds2wz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t02sh3cjhqhk5sqr2356elqhegnu68u-v20261001/router-6c76dd549c-ds2wz has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #7064	2102534053163962368
2026-09-22 12:15	PR #6364	2102371297248940032

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:08:48Z
  Ended:       2026-09-23T00:37:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e0ac9c-ae62-4294-a73a-42eb6033d5d4"
  Description: Pod ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-23T00:09:48Z
  Ended:       2026-09-23T00:37:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e0ac9c-ae62-4294-a73a-42eb6033d5d4"
  Description: Pod ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T17:19:59Z
  Ended:       2026-09-22T17:44:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="603dfbeb-4dca-4193-8523-f1040f307d47"
  Description: Pod ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-22T17:19:59Z
  Ended:       2026-09-22T17:48:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="603dfbeb-4dca-4193-8523-f1040f307d47"
  Description: Pod ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #7064	2102534053163962368
2026-09-22 16:02	PR #6503	2102428363070115840

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T00:19:55Z
  Ended:       2026-09-23T00:26:51Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T00:19:55Z
  Ended:       2026-09-23T00:25:51Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-23T00:19:55Z
  Ended:       2026-09-23T00:26:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-23T00:19:55Z
  Ended:       2026-09-23T00:25:51Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-23T00:19:56Z
  Ended:       2026-09-23T00:25:51Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T17:30:47Z
  Ended:       2026-09-22T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/services-j0115840", namespace="klusterlet-2t06amajdmngpo94r6274auvf1ibbeql", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t06amajdmngpo94r6274auvf1ibbeql/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-22T17:30:47Z
  Ended:       2026-09-22T17:37:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/services-j0115840", namespace="klusterlet-2t06amajdmngpo94r6274auvf1ibbeql", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t06amajdmngpo94r6274auvf1ibbeql/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:02	PR #7064	2102534053163962368
2026-09-22 16:02	PR #6503	2102428363070115840

ERROR CODE: Conflict; detail code ObjectIsDeletedButRecoverable

full failure pattern:

ERROR CODE: Conflict; detail code ObjectIsDeletedButRecoverable

Signal:

provision

none

Full failure examples (2)

time=2026-09-22T03:15:16.448Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert description="Step admin-api-cert\n Kind: CreateCertificate\n"
time=2026-09-22T03:15:16.448Z level=INFO msg="Creating certificate" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert vaultBaseUrl=https://aro-hcp-dev-svc-kv.vault.azure.net/ certificateName=admin-api-cert-ci01-j2691712 issuer=Self commonName=admin.j2691712.hcpsvc.osadev.cloud san=admin.j2691712.hcpsvc.osadev.cloud
time=2026-09-22T03:15:17.222Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert err="error running Create Certificate Step: failed to create certificate \"admin-api-cert-ci01-j2691712\" in vault \"https://aro-hcp-dev-svc-kv.vault.azure.net/\": POST https://aro-hcp-dev-svc-kv.vault.azure.net/certificates/admin-api-cert-ci01-j2691712/create\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Certificate admin-api-cert-ci01-j2691712 is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged.\",\n    \"innererror\": {\n      \"code\": \"ObjectIsDeletedButRecoverable\"\n    }\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-22T00:26:51.703Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert description="Step admin-api-cert\n Kind: CreateCertificate\n"
time=2026-09-22T00:26:51.703Z level=INFO msg="Creating certificate" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert vaultBaseUrl=https://aro-hcp-dev-svc-kv.vault.azure.net/ certificateName=admin-api-cert-ci01-j1796608 issuer=Self commonName=admin.j1796608.hcpsvc.osadev.cloud san=admin.j1796608.hcpsvc.osadev.cloud
time=2026-09-22T00:26:52.483Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert err="error running Create Certificate Step: failed to create certificate \"admin-api-cert-ci01-j1796608\" in vault \"https://aro-hcp-dev-svc-kv.vault.azure.net/\": POST https://aro-hcp-dev-svc-kv.vault.azure.net/certificates/admin-api-cert-ci01-j1796608/create\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Certificate admin-api-cert-ci01-j1796608 is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged.\",\n    \"innererror\": {\n      \"code\": \"ObjectIsDeletedButRecoverable\"\n    }\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/admin-api-cert	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 02:42	PR #7062	2102226956232691712
2026-09-22 00:02	PR #7042	2102186818521796608

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

full failure pattern:

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:46	PR #6889	2102424261724475392
2026-09-22 12:46	PR #6889	2102379019017129984

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (2)

time=2026-09-22T10:35:24.276Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T10:42:31.669Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T10:35:24.282791483Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T10:35:26.284841392Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T10:35:26.448708771Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T10:42:31.666852933Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"48630ADF8315F9F4\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j2444800-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT1M40.6270261S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:42:31.666918654Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"50E367DC062B740B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j2444800-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT1M39.3479577S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:42:31.666940194Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"96EF32884A17F314\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j2444800-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT4.1058994S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:42:31.666978395Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1E29754134C2A673\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j2444800-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:35:30.2867656Z\",\"duration\":\"PT5M11.8733849S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:42:31.667019736Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DD00B42016EFC1CA\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j2444800-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT3.4027868S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:42:31.667063427Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"3962D5B17452A0C8\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2\",\"resourceName\":\"ci01-j2444800-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:35:30.2867656Z\",\"duration\":\"PT5M8.2796787S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:42:31.667099327Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115335588388698\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m7.384317605s\"}\n exit status 1"

time=2026-09-22T10:31:08.961Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T10:38:47.425Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T10:31:08.979907747Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T10:31:11.06396002Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T10:31:11.22661971Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T10:38:47.423846978Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F61EEEEBCFA56868\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j4070272-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT1M39.7596603S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:38:47.423898879Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"8E07C9049ECB8DEF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j4070272-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT1M39.7909151S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:38:47.423911859Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"C8376C88B6FADF35\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j4070272-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT4.063355S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:38:47.42394617Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2DFDEDAE29CB0E2C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j4070272-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT3.9227798S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:38:47.423968521Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"04012E211BF318FE\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j4070272-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:31:16.3138337Z\",\"duration\":\"PT5M28.8419351S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:38:47.424009742Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F96D9B5A6875A606\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2\",\"resourceName\":\"ci01-j4070272-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:31:16.3138337Z\",\"duration\":\"PT5M25.0918247S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:38:47.424030272Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115338140513340\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m38.444136305s\"}\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 10:04	PR #7009	2102338247882444800
2026-09-22 09:46	PR #6439	2102333755334070272

cluster entered terminal Failed state after role assignment deployment

full failure pattern:

cluster entered terminal Failed state after role assignment deployment

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:46	PR #6889	2102424261724475392
2026-09-22 12:46	PR #6889	2102379019017129984

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-svhj5w"
Unexpected error:
    <*fmt.wrapError | 0xc0017e4720>: 
    DNS for route host agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-svhj5w"
Unexpected error:
    <*fmt.wrapError | 0xc0017e4720>: 
    DNS for route host agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:157]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc001120bc0>: 
    DNS for route host agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:157]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc001120bc0>: 
    DNS for route host agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 06:00	PR #7053	2102276754965008384
2026-09-22 05:59	PR #6948	2102276691408719872

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-zck558"
Unexpected error:
    <*fmt.wrapError | 0xc001112760>: 
    DNS for route host agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-zck558"
Unexpected error:
    <*fmt.wrapError | 0xc001112760>: 
    DNS for route host agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-84mf8p"
Unexpected error:
    <*fmt.wrapError | 0xc001186040>: 
    DNS for route host agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-84mf8p"
Unexpected error:
    <*fmt.wrapError | 0xc001186040>: 
    DNS for route host agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 17:15	PR #4878	2102448371858935808
2026-09-22 07:09	PR #6364	2102294129609281536

timeout during CreateHCPClusterAndWait; context deadline exceeded

full failure pattern:

timeout during CreateHCPClusterAndWait; context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000af31d0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-r6xxwmvr4vzb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-r6xxwmvr4vzb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000af31d0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-r6xxwmvr4vzb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-r6xxwmvr4vzb, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000a123f0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-57bkrhmpnrrx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-57bkrhmpnrrx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000a123f0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-57bkrhmpnrrx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-57bkrhmpnrrx, error: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:46	PR #6889	2102424261724475392
2026-09-22 12:46	PR #6889	2102379019017129984

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c521c0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-8ql5kd5f7tjm" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33",
      "name": "547703e0-0d1c-43cc-a185-feb70bc56c33",
      "status": "Failed",
      "startTime": "2026-09-22T16:36:27.4357733Z",
      "endTime": "2026-09-22T17:05:34.221694646Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c521c0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-8ql5kd5f7tjm" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33",
      "name": "547703e0-0d1c-43cc-a185-feb70bc56c33",
      "status": "Failed",
      "startTime": "2026-09-22T16:36:27.4357733Z",
      "endTime": "2026-09-22T17:05:34.221694646Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000d06700>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-b5tsprkj7fb4" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461",
      "name": "20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461",
      "status": "Failed",
      "startTime": "2026-09-22T16:36:09.86553733Z",
      "endTime": "2026-09-22T17:05:14.887023579Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000d06700>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-b5tsprkj7fb4" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461",
      "name": "20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461",
      "status": "Failed",
      "startTime": "2026-09-22T16:36:09.86553733Z",
      "endTime": "2026-09-22T17:05:14.887023579Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-r2ls78" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b566a0>: 
    failed to create HCP cluster cluster-zstream-4-22-r2ls78: failed waiting for cluster="cluster-zstream-4-22-r2ls78" in resourcegroup="rg-zstream-upgrade-4-22-sm79xt4bwmhx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3",
      "name": "55386486-26be-4228-bb22-1d85147aacf3",
      "status": "Failed",
      "startTime": "2026-09-22T16:36:16.148910931Z",
      "endTime": "2026-09-22T17:05:24.221837381Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-r2ls78" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b566a0>: 
    failed to create HCP cluster cluster-zstream-4-22-r2ls78: failed waiting for cluster="cluster-zstream-4-22-r2ls78" in resourcegroup="rg-zstream-upgrade-4-22-sm79xt4bwmhx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3",
      "name": "55386486-26be-4228-bb22-1d85147aacf3",
      "status": "Failed",
      "startTime": "2026-09-22T16:36:16.148910931Z",
      "endTime": "2026-09-22T17:05:24.221837381Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (56)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE can pause schedules to stop backup execution for an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 15:46	PR #6889	2102424261724475392

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d58a20>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-g7q8vnwmjvvk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a",
      "name": "ea0270f1-32ca-4b4d-a131-126d53d2800a",
      "status": "Failed",
      "startTime": "2026-09-22T14:06:37.339670093Z",
      "endTime": "2026-09-22T14:35:38.595765601Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d58a20>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-g7q8vnwmjvvk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a",
      "name": "ea0270f1-32ca-4b4d-a131-126d53d2800a",
      "status": "Failed",
      "startTime": "2026-09-22T14:06:37.339670093Z",
      "endTime": "2026-09-22T14:35:38.595765601Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010c0b80>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-qwx5tf2dtpk2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce",
      "name": "5e31a2de-3b83-48d4-a41a-2116a2e676ce",
      "status": "Failed",
      "startTime": "2026-09-22T13:32:01.635132061Z",
      "endTime": "2026-09-22T14:01:04.903065926Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010c0b80>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-qwx5tf2dtpk2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce",
      "name": "5e31a2de-3b83-48d4-a41a-2116a2e676ce",
      "status": "Failed",
      "startTime": "2026-09-22T13:32:01.635132061Z",
      "endTime": "2026-09-22T14:01:04.903065926Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-kbjcpq" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000f728e0>: 
    failed to create HCP cluster cluster-zstream-4-21-kbjcpq: failed waiting for cluster="cluster-zstream-4-21-kbjcpq" in resourcegroup="rg-zstream-upgrade-4-21-ftfjp89m25kk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9",
      "name": "3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9",
      "status": "Failed",
      "startTime": "2026-09-22T13:32:22.061338875Z",
      "endTime": "2026-09-22T14:01:24.909726517Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-kbjcpq" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000f728e0>: 
    failed to create HCP cluster cluster-zstream-4-21-kbjcpq: failed waiting for cluster="cluster-zstream-4-21-kbjcpq" in resourcegroup="rg-zstream-upgrade-4-21-ftfjp89m25kk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9",
      "name": "3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9",
      "status": "Failed",
      "startTime": "2026-09-22T13:32:22.061338875Z",
      "endTime": "2026-09-22T14:01:24.909726517Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (56)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE can pause schedules to stop backup execution for an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 12:46	PR #6889	2102379019017129984

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000fb01e0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-22ntvrtddssz" to finish creating: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1",
      "name": "242f24a5-8f64-4654-9b39-e30267e94ed1",
      "status": "Failed",
      "startTime": "2026-09-22T13:01:52.830008497Z",
      "endTime": "2026-09-22T13:30:57.820386496Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 21m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000fb01e0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-22ntvrtddssz" to finish creating: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1",
      "name": "242f24a5-8f64-4654-9b39-e30267e94ed1",
      "status": "Failed",
      "startTime": "2026-09-22T13:01:52.830008497Z",
      "endTime": "2026-09-22T13:30:57.820386496Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 21m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 12:15	PR #6364	2102371297248940032

at least one schedule should have a LastBackupTime

full failure pattern:

at least one schedule should have a LastBackupTime

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.007s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.007s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE can pause schedules to stop backup execution for an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 16:02	PR #6503	2102428363070115840

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:329]: failed to get admin REST config for cluster np-noedge-7z925l
Unexpected error:
    <*fmt.wrapError | 0xc00077d780>: 
    failed waiting for hcpCluster="np-noedge-7z925l" in resourcegroup="rg-np-noedge-7z925l-xwhx4zwbzvsx" to finish getting creds: GET https://rp.j7511040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:329]: failed to get admin REST config for cluster np-noedge-7z925l
Unexpected error:
    <*fmt.wrapError | 0xc00077d780>: 
    failed waiting for hcpCluster="np-noedge-7z925l" in resourcegroup="rg-np-noedge-7z925l-xwhx4zwbzvsx" to finish getting creds: GET https://rp.j7511040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 11:04	PR #6092	2102353447767511040

ERROR CODE: DeploymentFailed; detail code ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity; detail message Too many Federated Identity Credentials are written ...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity; detail message Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.managedidentity/userassignedidentities/fleet'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported.

Signal:

provision

none

Full failure examples (1)

time=2026-09-22T04:51:02.686Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-09-22T04:51:04.267Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-09-22T04:51:06.089Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=96622666cca941a93b9a8651dce6827ae1f27fb7972b41f9caa78871db9bc985 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9675136-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F96622666cca941a93b9a8651dce6827ae1f27fb7972b41f9caa78871db9bc985
time=2026-09-22T05:04:09.339Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9675136-svc/providers/Microsoft.Resources/deployments/96622666cca941a93b9a8651dce6827ae1f27fb7972b41f9caa78871db9bc985/operationStatuses/08584115542204326289\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9675136-svc/providers/Microsoft.Resources/deployments/cluster-olgmgzuso436u\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity\\\",\\r\\n            \\\"message\\\": \\\"Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9675136-svc/providers/microsoft.managedidentity/userassignedidentities/fleet'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 04:31	PR #6516	2102254439539675136

ERROR CODE: DeploymentFailed; detail code ReconcileVMSSAgentPoolFailed; provider Microsoft.ContainerService

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ReconcileVMSSAgentPoolFailed; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (1)

time=2026-09-22T00:28:42.028Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-22T00:31:19.046Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T00:28:42.035099926Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-22T00:28:44.199184892Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T00:28:44.294775263Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T00:31:19.041004649Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B5981CAA76057630\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5677184-mgmt-1\",\"resourceName\":\"ci01-j5677184-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T00:28:49.2033491Z\",\"duration\":\"PT2M28.4887595S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"ReconcileVMSSAgentPoolFailed\",\"message\":\"Internal server error\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T00:31:19.041107186Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115699609930281\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5677184-mgmt-1\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"ReconcileVMSSAgentPoolFailed\\\",\\n            \\\"message\\\": \\\"Internal server error\\\"\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"2m37.00601594s\"}\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 00:02	PR #5846	2102186730835677184

client rate limiter Wait returned an error: context deadline exceeded

full failure pattern:

client rate limiter Wait returned an error: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0025d8000>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0025d8000>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 13:12	PR #6688	2102385490215833600

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001f04078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xvhgrrnh7hjx" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9",
      "name": "177d426f-dd32-4799-8b9b-eb1aaa182ca9",
      "status": "Failed",
      "startTime": "2026-09-22T23:54:44.577772615Z",
      "endTime": "2026-09-23T00:18:50.338452518Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001f04078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xvhgrrnh7hjx" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9",
      "name": "177d426f-dd32-4799-8b9b-eb1aaa182ca9",
      "status": "Failed",
      "startTime": "2026-09-22T23:54:44.577772615Z",
      "endTime": "2026-09-23T00:18:50.338452518Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 22:46	PR #6359	2102530027982688256

alert [hcp] userJourneySwiftRouterAvailability1h5m fired

full failure pattern:

alert [hcp] userJourneySwiftRouterAvailability1h5m fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T23:53:49Z
  Ended:       2026-09-23T00:03:49Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j2688256-mgmt-1", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2688256/providers/microsoft.monitor/accounts/hcps-j2688256", region="canadacentral", severity="3", short_window="5m", slo="swift-router-availability"
  Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneySwiftRouterAvailability1h5m does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 22:46	PR #6359	2102530027982688256

failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded

full failure pattern:

failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-09-22T07:21:38.848Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-09-22T07:21:40.354Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-09-22T07:21:41.548Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=d03eff9eca8ce6668b3735feb0fed36048a18f0177222fc337f65cc4e325d887 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4723968-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd03eff9eca8ce6668b3735feb0fed36048a18f0177222fc337f65cc4e325d887
time=2026-09-22T07:51:38.563Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 06:52	PR #7060	2102289975184723968

Environment: INT

Failure patterns: 2 · Failures matched: 3 · Total runs: 3 · Runs affected: 1

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000914940>: 
    failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "status": "Failed",
      "startTime": "2026-09-22T21:22:56.593640254Z",
      "endTime": "2026-09-22T21:51:58.372098632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000914940>: 
    failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "status": "Failed",
      "startTime": "2026-09-22T21:22:56.593640254Z",
      "endTime": "2026-09-22T21:51:58.372098632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc00102a600>: 
    failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "status": "Failed",
      "startTime": "2026-09-22T21:07:20.636049246Z",
      "endTime": "2026-09-22T21:36:27.958431734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc00102a600>: 
    failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "status": "Failed",
      "startTime": "2026-09-22T21:07:20.636049246Z",
      "endTime": "2026-09-22T21:36:27.958431734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 19:50	n/a	2102492733552726016

failed to locate cilium chart: Get "<url>": dial tcp <ip>: connect: network is unreachable

full failure pattern:

failed to locate cilium chart: Get "<url>": dial tcp <ip>: connect: network is unreachable

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm
Unexpected error:
    <*fmt.wrapError | 0xc0008581e0>: 
    failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm
Unexpected error:
    <*fmt.wrapError | 0xc0008581e0>: 
    failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 19:50	n/a	2102492733552726016

Environment: PROD

Failure patterns: 7 · Failures matched: 14 · Total runs: 2 · Runs affected: 1

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002790330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rnbkplrwl7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002790330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rnbkplrwl7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00148e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-7dfqps65dplq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00148e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-7dfqps65dplq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0025133b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cxzbgzk6bm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0025133b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cxzbgzk6bm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571620114432

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001385da0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4tjrbb" resourcegroup="rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4tjrbb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001385da0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4tjrbb" resourcegroup="rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4tjrbb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc0010981a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-x28k9h" in resourcegroup="rg-np-skip-minor-x28k9h-x2cmwhr29p2s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-x28k9h-x2cmwhr29p2s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-x28k9h/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc0010981a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-x28k9h" in resourcegroup="rg-np-skip-minor-x28k9h-x2cmwhr29p2s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-x28k9h-x2cmwhr29p2s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-x28k9h/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k"
Unexpected error:
    <*fmt.wrapError | 0xc000ce4120>: 
    failed to create NodePool np-6gs84k: failed starting nodepool creation "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" in resourcegroup="rg-nightly-4-20-cv5qj2kl686k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cv5qj2kl686k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-6gs84k/nodePools/np-6gs84k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k"
Unexpected error:
    <*fmt.wrapError | 0xc000ce4120>: 
    failed to create NodePool np-6gs84k: failed starting nodepool creation "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" in resourcegroup="rg-nightly-4-20-cv5qj2kl686k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cv5qj2kl686k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-6gs84k/nodePools/np-6gs84k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571620114432

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ef1980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dkldtr9wz7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "name": "baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "status": "Failed",
      "startTime": "2026-09-22T23:22:56.369741564Z",
      "endTime": "2026-09-22T23:42:02.437458738Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ef1980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dkldtr9wz7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "name": "baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "status": "Failed",
      "startTime": "2026-09-22T23:22:56.369741564Z",
      "endTime": "2026-09-22T23:42:02.437458738Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should create an HCP cluster and validate TLS certificates	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571620114432

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00103f360>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00103f360>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571620114432

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000fb5908>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-210852, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000fb5908>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-210852, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571620114432

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster servi...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-ctwh8htmlpcs/cluster-nightly-5-1-v48g4k should provision
Unexpected error:
    <*fmt.wrapError | 0xc00150a700>: 
    failed to create HCP cluster cluster-nightly-5-1-v48g4k: failed waiting for cluster="cluster-nightly-5-1-v48g4k" in resourcegroup="rg-nightly-5-1-ctwh8htmlpcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "name": "219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "status": "Failed",
      "startTime": "2026-09-22T23:38:52.209443302Z",
      "endTime": "2026-09-23T00:07:57.16268573Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-ctwh8htmlpcs/cluster-nightly-5-1-v48g4k should provision
Unexpected error:
    <*fmt.wrapError | 0xc00150a700>: 
    failed to create HCP cluster cluster-nightly-5-1-v48g4k: failed waiting for cluster="cluster-nightly-5-1-v48g4k" in resourcegroup="rg-nightly-5-1-ctwh8htmlpcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "name": "219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "status": "Failed",
      "startTime": "2026-09-22T23:38:52.209443302Z",
      "endTime": "2026-09-23T00:07:57.16268573Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571620114432

cluster CA data must contain exactly one PEM block

full failure pattern:

cluster CA data must contain exactly one PEM block

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571620114432

Environment: STG

Failure patterns: 2 · Failures matched: 108 · Total runs: 2 · Runs affected: 2

Failure pattern

Failed at

Trend

Also in

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHat...

full failure pattern:

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000698d00>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-9ggtk5tbv4fl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-9ggtk5tbv4fl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000698d00>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-9ggtk5tbv4fl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-9ggtk5tbv4fl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0006784e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l2fgd6zqtf9h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l2fgd6zqtf9h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0006784e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l2fgd6zqtf9h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l2fgd6zqtf9h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101bd80>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-jnltm5nznwcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-jnltm5nznwcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101bd80>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-jnltm5nznwcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-jnltm5nznwcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (107)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to delete an HCP cluster whose managed identities were deleted first	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to list available HCP OpenShift versions and validate response content	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with back-level version 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create node pools with ARM64-based VMs	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to delete a nodepool	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to delete an HCP cluster whose managed identities were deleted first	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to list available HCP OpenShift versions and validate response content	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-22 23:00	n/a	2102533571657863168
2026-09-22 02:00	n/a	2102216478680944640

failed to create HCP cluster <cluster>: failed starting cluster creation "<cluster>" in resourcegroup="<resource-group>": Put "<url>": stream error: stream ID 37; CANCEL; received...

full failure pattern:

failed to create HCP cluster <cluster>: failed starting cluster creation "<cluster>" in resourcegroup="<resource-group>": Put "<url>": stream error: stream ID 37; CANCEL; received from peer

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0013940e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0013940e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-22 02:00	n/a	2102216478680944640