CIHealth Failure Patterns

Environment: DEV

Failure patterns: 47 · Failures matched: 232 · Total runs: 92 · Runs affected: 80

Failure pattern

Failed at

Trend

Also in

VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4

full failure pattern:

VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc001389008>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.22.13) failed: no version in clusterversion/version status.history is greater than initial "4.22.13"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc001389008>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.22.13) failed: no version in clusterversion/version status.history is greater than initial "4.22.13"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c984b0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.33) failed: no version in clusterversion/version status.history is greater than initial "4.21.33"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c984b0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.33) failed: no version in clusterversion/version status.history is greater than initial "4.21.33"
    ...

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 15:43	PR #6577	2102066961654484992

failed to get region configuration for region "westus3": the cloud public is not found in the config

full failure pattern:

failed to get region configuration for region "westus3": the cloud public is not found in the config

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:44]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc00011d9e0>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:44]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc00011d9e0>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:68]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc0014eed00>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:68]: failed to load service config
Unexpected error:
    <*fmt.wrapError | 0xc0014eed00>: 
    failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	AMA Metrics service metrics from SVC and MGMT clusters should be present in Azure Monitor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	KSM HCP Metrics kube_node_info metrics should be present in Azure Monitor for the happy-path cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 08:52	PR #7009	2101957660935983104

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T18:04:56.091Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T18:12:34.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T18:04:56.09869494Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-09-21T18:04:58.120073331Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T18:04:58.527982819Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-21T18:12:34.652011621Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"89CF033DB1DF73D0\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j7635328-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT1M56.3405526S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:12:34.652086242Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A0DB23EC9205187A\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j7635328-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT1M39.5985567S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:12:34.652107413Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"977918EE1130A569\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j7635328-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT4.3101609S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:12:34.652152984Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"5835781ECA9C15E0\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j7635328-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:05:03.7378518Z\",\"duration\":\"PT5M26.5601897S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:12:34.652204585Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E787274800217612\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j7635328-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:10:26.0503797Z\",\"duration\":\"PT4.0601636S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:12:34.652246526Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B468A793AB498001\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2\",\"resourceName\":\"ci01-j7635328-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:05:03.7378518Z\",\"duration\":\"PT5M22.2477188S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:12:34.652279217Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7635328-mgmt-2\",\"clusterName\":\"ci01-j7635328-mgmt-2\",\"region\":\"westus3\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115929865651702\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7635328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7635328-mgmt-2/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 32 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m38.553591047s\"}\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 16:54	PR #7042	2102083142897635328

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j4808064-mgmt-1","clusterName":"ci01-j480806

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T10:06:30.701Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T10:36:30.417Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T10:06:30.708972948Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4808064-mgmt-1\",\"clusterName\":\"ci01-j4808064-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-09-21T10:06:33.118843713Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4808064-mgmt-1\",\"clusterName\":\"ci01-j4808064-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T10:06:33.444184017Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4808064-mgmt-1\",\"clusterName\":\"ci01-j4808064-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 09:34	PR #7003	2101968322814808064

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j8977280-mgmt-1","clusterName":"ci01-j897728

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T10:03:47.924Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T10:33:47.694Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T10:03:47.930202871Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8977280-mgmt-1\",\"clusterName\":\"ci01-j8977280-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-21T10:03:49.993282994Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8977280-mgmt-1\",\"clusterName\":\"ci01-j8977280-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T10:03:50.275505506Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8977280-mgmt-1\",\"clusterName\":\"ci01-j8977280-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 09:41	PR #7031	2101970018378977280

error: failed to create clusterrolebinding: etcdserver: request timed out, possibly due to previous leader failure

full failure pattern:

error: failed to create clusterrolebinding: etcdserver: request timed out, possibly due to previous leader failure

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:293]: failed to create cluster role binding for external auth via VM
Unexpected error:
    <*errors.errorString | 0xc000aafea0>: 
    error: failed to create clusterrolebinding: etcdserver: request timed out, possibly due to previous leader failure
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:293]: failed to create cluster role binding for external auth via VM
Unexpected error:
    <*errors.errorString | 0xc000aafea0>: 
    error: failed to create clusterrolebinding: etcdserver: request timed out, possibly due to previous leader failure
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 14:59	PR #7031	2102066654929227776

failed to load service config: failed to get region configuration for region "westus3": the cloud public is not found in the config

full failure pattern:

failed to load service config: failed to get region configuration for region "westus3": the cloud public is not found in the config

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc00039e080>: 
    failed to load service config: failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc00039e080>: 
    failed to load service config: failed to get region configuration for region "westus3": the cloud public is not found in the config
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 08:52	PR #7009	2101957660935983104

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000b0e870>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pg24gmdfnmzw" to finish deleting: GET https://rp.j0434432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7836cbe1-bcbe-4646-898f-78e819a0d98f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7836cbe1-bcbe-4646-898f-78e819a0d98f",
      "name": "7836cbe1-bcbe-4646-898f-78e819a0d98f",
      "status": "Failed",
      "startTime": "2026-09-21T23:35:23.137655617Z",
      "endTime": "2026-09-21T23:59:32.105878829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svmdju7l81957j5gpbargner8e51jae still exists (deletion dispatched at 2026-09-21T23:35:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000b0e870>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pg24gmdfnmzw" to finish deleting: GET https://rp.j0434432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7836cbe1-bcbe-4646-898f-78e819a0d98f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7836cbe1-bcbe-4646-898f-78e819a0d98f",
      "name": "7836cbe1-bcbe-4646-898f-78e819a0d98f",
      "status": "Failed",
      "startTime": "2026-09-21T23:35:23.137655617Z",
      "endTime": "2026-09-21T23:59:32.105878829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svmdju7l81957j5gpbargner8e51jae still exists (deletion dispatched at 2026-09-21T23:35:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f3a090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-9kb6b8kvhn7s" to finish deleting: GET https://rp.j0434432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/825dbd39-48b3-4fef-8b8e-49148be72c72
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/825dbd39-48b3-4fef-8b8e-49148be72c72",
      "name": "825dbd39-48b3-4fef-8b8e-49148be72c72",
      "status": "Failed",
      "startTime": "2026-09-21T23:21:12.831453866Z",
      "endTime": "2026-09-21T23:45:12.878649358Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svmegsg25m4rvg6v9g6bgt2bkfpnu3j still exists (deletion dispatched at 2026-09-21T23:21:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f3a090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-9kb6b8kvhn7s" to finish deleting: GET https://rp.j0434432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/825dbd39-48b3-4fef-8b8e-49148be72c72
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/825dbd39-48b3-4fef-8b8e-49148be72c72",
      "name": "825dbd39-48b3-4fef-8b8e-49148be72c72",
      "status": "Failed",
      "startTime": "2026-09-21T23:21:12.831453866Z",
      "endTime": "2026-09-21T23:45:12.878649358Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svmegsg25m4rvg6v9g6bgt2bkfpnu3j still exists (deletion dispatched at 2026-09-21T23:21:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c52168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-pgjxph86z52p" to finish deleting: GET https://rp.j2480256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50a0df80-c8f5-49a6-a2ef-c644f0089148
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50a0df80-c8f5-49a6-a2ef-c644f0089148",
      "name": "50a0df80-c8f5-49a6-a2ef-c644f0089148",
      "status": "Failed",
      "startTime": "2026-09-21T22:50:27.873366155Z",
      "endTime": "2026-09-21T23:14:33.586111357Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svlspsboe6grnibds7i6q05t212qj6b still exists (deletion dispatched at 2026-09-21T22:50:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c52168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-pgjxph86z52p" to finish deleting: GET https://rp.j2480256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50a0df80-c8f5-49a6-a2ef-c644f0089148
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50a0df80-c8f5-49a6-a2ef-c644f0089148",
      "name": "50a0df80-c8f5-49a6-a2ef-c644f0089148",
      "status": "Failed",
      "startTime": "2026-09-21T22:50:27.873366155Z",
      "endTime": "2026-09-21T23:14:33.586111357Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svlspsboe6grnibds7i6q05t212qj6b still exists (deletion dispatched at 2026-09-21T22:50:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (21)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	37
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	7
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1

Affected runs (49)

Date (UTC)	Associated PR	Prow job
2026-09-21 22:11	PR #7057	2102158779100434432
2026-09-21 21:28	PR #5846	2102147905392480256
2026-09-21 20:53	PR #7053	2102139231773134848
2026-09-21 20:48	PR #6364	2102137984487788544
2026-09-21 20:00	PR #7050	2102126077554987008
2026-09-21 20:00	PR #6969	2102125829403185152
2026-09-21 19:46	PR #6945	2102122240874123264
2026-09-21 19:38	PR #6925	2102120305865527296
2026-09-21 19:22	PR #7020	2102116189634301952
2026-09-21 19:15	PR #7054	2102114596662808576
2026-09-21 18:01	PR #7044	2102095923193581568
2026-09-21 18:01	PR #7033	2102095920672804864
2026-09-21 18:01	PR #6886	2102095915643834368
2026-09-21 18:01	PR #6948	2102095907259420672
2026-09-21 17:04	PR #7048	2102086108664827904
2026-09-21 16:50	PR #6359	2102079067309215744
2026-09-21 16:38	PR #7049	2102076044226859008
2026-09-21 16:24	PR #5846	2102072370310156288
2026-09-21 15:50	PR #6945	2102076823893446656
2026-09-21 15:45	PR #6688	2102065610849521664
2026-09-21 15:22	PR #6891	2102071948841324544
2026-09-21 15:07	PR #6965	2102073111963766784
2026-09-21 15:00	PR #6886	2102058858829582336
2026-09-21 13:23	PR #7001	2102025876047663104
2026-09-21 13:20	PR #7009	2102025100818649088
2026-09-21 13:00	PR #6948	2102020232922009600
2026-09-21 13:00	PR #7033	2102020127040999424
2026-09-21 12:01	PR #6886	2102005383924027392
2026-09-21 12:00	PR #7044	2102005289971617792
2026-09-21 12:00	PR #6995	2102005251379826688
2026-09-21 11:46	PR #6965	2102001525193707520
2026-09-21 11:43	PR #7036	2102000884778012672
2026-09-21 11:34	PR #7016	2101998376915570688
2026-09-21 09:24	PR #7044	2101965713999663104
2026-09-21 09:19	PR #6948	2101964585778024448
2026-09-21 09:15	PR #7034	2101963613219590144
2026-09-21 09:00	PR #7042	2101959649765888000
2026-09-21 08:52	PR #7009	2101957660935983104
2026-09-21 08:24	PR #6886	2101950652551794688
2026-09-21 06:45	PR #7042	2101925644194549760
2026-09-21 05:59	PR #5724	2101914215290769408
2026-09-21 05:49	PR #6948	2101911584593940480
2026-09-21 05:07	PR #6439	2101901188189392896
2026-09-21 04:29	PR #7032	2101892420403728384
2026-09-21 03:54	PR #6092	2101882720345919488
2026-09-21 03:02	PR #6225	2101869717533757440
2026-09-21 02:16	PR #7036	2101858124586553344
2026-09-21 02:00	PR #6965	2101853992668631040
2026-09-21 01:27	PR #7000	2101845814870544384

alert [svc] FrontendPathMedianLatency fired

full failure pattern:

alert [svc] FrontendPathMedianLatency fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T00:44:59Z
  Ended:       2026-09-22T01:17:56Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9104768-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9104768-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T23:02:39Z
  Ended:       2026-09-21T23:30:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2480256-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} has exceeded 250ms over the past 30 minutes (cluster ci01-j2480256-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T22:19:36Z
  Ended:       2026-09-21T22:27:37Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3134848-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3134848-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire	20

Affected runs (20)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:45	PR #6359	2102182452419104768
2026-09-21 21:28	PR #5846	2102147905392480256
2026-09-21 20:53	PR #7053	2102139231773134848
2026-09-21 20:48	PR #6364	2102137984487788544
2026-09-21 20:00	PR #7050	2102126077554987008
2026-09-21 20:00	PR #7042	2102125896348471296
2026-09-21 19:24	PR #5846	2102116740375777280
2026-09-21 19:15	PR #7054	2102114596662808576
2026-09-21 16:38	PR #7049	2102076044226859008
2026-09-21 16:38	PR #6925	2102075143474909184
2026-09-21 16:24	PR #5846	2102072370310156288
2026-09-21 15:50	PR #6945	2102076823893446656
2026-09-21 15:22	PR #6891	2102071948841324544
2026-09-21 12:00	PR #7044	2102005289971617792
2026-09-21 11:46	PR #6965	2102001525193707520
2026-09-21 09:22	PR #7030	2101965282229620736
2026-09-21 06:45	PR #7042	2101925644194549760
2026-09-21 05:49	PR #6948	2101911584593940480
2026-09-21 02:16	PR #7036	2101858124586553344
2026-09-21 01:27	PR #7000	2101845814870544384

alert [svc] ServiceMemoryDrift fired

full failure pattern:

alert [svc] ServiceMemoryDrift fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-21T23:25:14Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j2480256-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-spg6n", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-spg6n (namespace arobit) on cluster ci01-j2480256-svc has averaged 126.7% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-21T22:52:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4559488-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-wmt7h", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-wmt7h (namespace arobit) on cluster ci01-j4559488-svc has averaged 126.9% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-21T21:29:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4123264-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-m4kdz", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-m4kdz (namespace arobit) on cluster ci01-j4123264-svc has averaged 127.7% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-21T21:30:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4123264-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-58zl4", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-58zl4 (namespace arobit) on cluster ci01-j4123264-mgmt-2 has averaged 122% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire	17

Affected runs (17)

Date (UTC)	Associated PR	Prow job
2026-09-21 21:28	PR #5846	2102147905392480256
2026-09-21 21:00	PR #7044	2102140947914559488
2026-09-21 19:46	PR #6945	2102122240874123264
2026-09-21 18:01	PR #6886	2102095915643834368
2026-09-21 18:01	PR #6948	2102095907259420672
2026-09-21 17:04	PR #7048	2102086108664827904
2026-09-21 16:50	PR #6359	2102079067309215744
2026-09-21 16:24	PR #5846	2102072370310156288
2026-09-21 15:45	PR #6688	2102065610849521664
2026-09-21 15:43	PR #6577	2102066961654484992
2026-09-21 11:46	PR #6965	2102001525193707520
2026-09-21 09:19	PR #6948	2101964585778024448
2026-09-21 08:52	PR #7009	2101957660935983104
2026-09-21 04:01	PR #7000	2101884774271422464
2026-09-21 03:02	PR #6225	2101869717533757440
2026-09-21 02:16	PR #7036	2101858124586553344
2026-09-21 02:00	PR #6965	2101853992668631040

ERROR CODE: Conflict; detail code ObjectIsDeletedButRecoverable

full failure pattern:

ERROR CODE: Conflict; detail code ObjectIsDeletedButRecoverable

Signal:

provision

none

Full failure examples (3)

time=2026-09-21T23:34:09.997Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert description="Step admin-api-cert\n Kind: CreateCertificate\n"
time=2026-09-21T23:34:09.998Z level=INFO msg="Creating certificate" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert vaultBaseUrl=https://aro-hcp-dev-svc-kv.vault.azure.net/ certificateName=admin-api-cert-ci01-j2953216 issuer=Self commonName=admin.j2953216.hcpsvc.osadev.cloud san=admin.j2953216.hcpsvc.osadev.cloud
time=2026-09-21T23:34:10.858Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert err="error running Create Certificate Step: failed to create certificate \"admin-api-cert-ci01-j2953216\" in vault \"https://aro-hcp-dev-svc-kv.vault.azure.net/\": POST https://aro-hcp-dev-svc-kv.vault.azure.net/certificates/admin-api-cert-ci01-j2953216/create\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Certificate admin-api-cert-ci01-j2953216 is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged.\",\n    \"innererror\": {\n      \"code\": \"ObjectIsDeletedButRecoverable\"\n    }\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-21T18:34:41.812Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert description="Step admin-api-cert\n Kind: CreateCertificate\n"
time=2026-09-21T18:34:41.812Z level=INFO msg="Creating certificate" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert vaultBaseUrl=https://aro-hcp-dev-svc-kv.vault.azure.net/ certificateName=admin-api-cert-ci01-j2537728 issuer=Self commonName=admin.j2537728.hcpsvc.osadev.cloud san=admin.j2537728.hcpsvc.osadev.cloud
time=2026-09-21T18:34:42.532Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert err="error running Create Certificate Step: failed to create certificate \"admin-api-cert-ci01-j2537728\" in vault \"https://aro-hcp-dev-svc-kv.vault.azure.net/\": POST https://aro-hcp-dev-svc-kv.vault.azure.net/certificates/admin-api-cert-ci01-j2537728/create\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Certificate admin-api-cert-ci01-j2537728 is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged.\",\n    \"innererror\": {\n      \"code\": \"ObjectIsDeletedButRecoverable\"\n    }\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-21T18:25:15.815Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert description="Step admin-api-cert\n Kind: CreateCertificate\n"
time=2026-09-21T18:25:15.815Z level=INFO msg="Creating certificate" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert vaultBaseUrl=https://aro-hcp-dev-svc-kv.vault.azure.net/ certificateName=admin-api-cert-ci01-j0667008 issuer=Self commonName=admin.j0667008.hcpsvc.osadev.cloud san=admin.j0667008.hcpsvc.osadev.cloud
time=2026-09-21T18:25:16.432Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert err="error running Create Certificate Step: failed to create certificate \"admin-api-cert-ci01-j0667008\" in vault \"https://aro-hcp-dev-svc-kv.vault.azure.net/\": POST https://aro-hcp-dev-svc-kv.vault.azure.net/certificates/admin-api-cert-ci01-j0667008/create\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Certificate admin-api-cert-ci01-j0667008 is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged.\",\n    \"innererror\": {\n      \"code\": \"ObjectIsDeletedButRecoverable\"\n    }\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/admin-api-cert	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:11	PR #7042	2102174018562953216
2026-09-21 18:18	PR #7055	2102100127102537728
2026-09-21 18:08	PR #7020	2102097561530667008
2026-09-21 17:27	PR #7050	2102089024922456064
2026-09-21 14:00	PR #6965	2102036202222784512
2026-09-21 04:41	PR #6948	2101894657490292736

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-d99twcmkwh84/cluster-candidate-5-0-8br7x9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001010280>: 
    failed to create HCP cluster cluster-candidate-5-0-8br7x9: failed waiting for cluster="cluster-candidate-5-0-8br7x9" in resourcegroup="rg-candidate-5-0-d99twcmkwh84" to finish creating: GET https://rp.j5527296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0799c7ba-4b49-4034-8d8b-2071f01c8afd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0799c7ba-4b49-4034-8d8b-2071f01c8afd",
      "name": "0799c7ba-4b49-4034-8d8b-2071f01c8afd",
      "status": "Failed",
      "startTime": "2026-09-21T20:26:31.347180566Z",
      "endTime": "2026-09-21T20:55:38.332601158Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-d99twcmkwh84/cluster-candidate-5-0-8br7x9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001010280>: 
    failed to create HCP cluster cluster-candidate-5-0-8br7x9: failed waiting for cluster="cluster-candidate-5-0-8br7x9" in resourcegroup="rg-candidate-5-0-d99twcmkwh84" to finish creating: GET https://rp.j5527296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0799c7ba-4b49-4034-8d8b-2071f01c8afd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0799c7ba-4b49-4034-8d8b-2071f01c8afd",
      "name": "0799c7ba-4b49-4034-8d8b-2071f01c8afd",
      "status": "Failed",
      "startTime": "2026-09-21T20:26:31.347180566Z",
      "endTime": "2026-09-21T20:55:38.332601158Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc00151cd50>: 
    GET https://rp.j3834368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8dbab85-98a4-49d4-94ad-b689a0da487b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8dbab85-98a4-49d4-94ad-b689a0da487b",
      "name": "e8dbab85-98a4-49d4-94ad-b689a0da487b",
      "status": "Failed",
      "startTime": "2026-09-21T18:52:59.240619957Z",
      "endTime": "2026-09-21T19:22:07.757736289Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc00151cd50>: 
    GET https://rp.j3834368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8dbab85-98a4-49d4-94ad-b689a0da487b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8dbab85-98a4-49d4-94ad-b689a0da487b",
      "name": "e8dbab85-98a4-49d4-94ad-b689a0da487b",
      "status": "Failed",
      "startTime": "2026-09-21T18:52:59.240619957Z",
      "endTime": "2026-09-21T19:22:07.757736289Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000b68880>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-t9sfgdtzmdz6" to finish creating: GET https://rp.j0999424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a83df19f-3103-42e8-8de4-e77b74ab884a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a83df19f-3103-42e8-8de4-e77b74ab884a",
      "name": "a83df19f-3103-42e8-8de4-e77b74ab884a",
      "status": "Failed",
      "startTime": "2026-09-21T14:10:33.153956692Z",
      "endTime": "2026-09-21T14:39:36.608452036Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000b68880>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-t9sfgdtzmdz6" to finish creating: GET https://rp.j0999424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a83df19f-3103-42e8-8de4-e77b74ab884a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a83df19f-3103-42e8-8de4-e77b74ab884a",
      "name": "a83df19f-3103-42e8-8de4-e77b74ab884a",
      "status": "Failed",
      "startTime": "2026-09-21T14:10:33.153956692Z",
      "endTime": "2026-09-21T14:39:36.608452036Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-21 19:38	PR #6925	2102120305865527296
2026-09-21 18:01	PR #6886	2102095915643834368
2026-09-21 13:00	PR #7033	2102020127040999424
2026-09-21 12:00	PR #7044	2102005289971617792

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T20:49:23Z
  Ended:       2026-09-21T21:18:25Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5527296-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5527296/providers/microsoft.monitor/accounts/hcps-j5527296", namespace="ocm-arohcpci01-2svk9tvinsjpvsqf4f95mmif44fb0pv7-v4j2g0l9y3n4g2w", pod="arohcp-monitor-kube-state-metrics-7955558448-qjq7d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2svk9tvinsjpvsqf4f95mmif44fb0pv7-v4j2g0l9y3n4g2w/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T21:00:23Z
  Ended:       2026-09-21T21:14:21Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5527296-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5527296/providers/microsoft.monitor/accounts/hcps-j5527296", namespace="ocm-arohcpci01-2svk9tvinsjpvsqf4f95mmif44fb0pv7-v4j2g0l9y3n4g2w", pod="arohcp-monitor-kube-state-metrics-7955558448-qjq7d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2svk9tvinsjpvsqf4f95mmif44fb0pv7-v4j2g0l9y3n4g2w/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T19:13:35Z
  Ended:       2026-09-21T19:47:40Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3834368-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3834368/providers/microsoft.monitor/accounts/hcps-j3834368", namespace="ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f", pod="arohcp-monitor-kube-state-metrics-7955558448-hlfnw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T19:13:35Z
  Ended:       2026-09-21T19:47:40Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3834368-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3834368/providers/microsoft.monitor/accounts/hcps-j3834368", namespace="ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f", pod="arohcp-monitor-kube-state-metrics-7955558448-hlfnw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T14:32:16Z
  Ended:       2026-09-21T14:58:07Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0999424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0999424/providers/microsoft.monitor/accounts/hcps-j0999424", namespace="ocm-arohcpci01-2svepmb9pe39k53ku2oiti573ipu6mda-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7955558448-mbr7z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2svepmb9pe39k53ku2oiti573ipu6mda-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T14:32:16Z
  Ended:       2026-09-21T15:00:11Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0999424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0999424/providers/microsoft.monitor/accounts/hcps-j0999424", namespace="ocm-arohcpci01-2svepmb9pe39k53ku2oiti573ipu6mda-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7955558448-mbr7z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2svepmb9pe39k53ku2oiti573ipu6mda-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-21 19:38	PR #6925	2102120305865527296
2026-09-21 18:01	PR #6886	2102095915643834368
2026-09-21 13:00	PR #7033	2102020127040999424
2026-09-21 12:00	PR #7044	2102005289971617792

alert [svc] FrontendPathLatency fired

full failure pattern:

alert [svc] FrontendPathLatency fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T18:10:16Z
  Ended:       2026-09-21T18:24:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j1324544-svc", component="frontend", method="delete", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j1324544-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T18:05:55Z
  Ended:       2026-09-21T18:11:41Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j3766784-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j3766784-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T06:15:25Z
  Ended:       2026-09-21T06:49:22Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9392896-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9392896-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-21 15:22	PR #6891	2102071948841324544
2026-09-21 15:07	PR #6965	2102073111963766784
2026-09-21 05:07	PR #6439	2101901188189392896
2026-09-21 02:00	PR #6965	2101853992668631040

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-98vcbn"
Unexpected error:
    <*fmt.wrapError | 0xc001510f00>: 
    DNS for route host agnhost-e2e-sample-app-cwflx.apps.aro.m5g7y3m1z1w6o1g.oqts.j9104768.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-cwflx.apps.aro.m5g7y3m1z1w6o1g.oqts.j9104768.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cwflx.apps.aro.m5g7y3m1z1w6o1g.oqts.j9104768.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-98vcbn"
Unexpected error:
    <*fmt.wrapError | 0xc001510f00>: 
    DNS for route host agnhost-e2e-sample-app-cwflx.apps.aro.m5g7y3m1z1w6o1g.oqts.j9104768.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-cwflx.apps.aro.m5g7y3m1z1w6o1g.oqts.j9104768.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cwflx.apps.aro.m5g7y3m1z1w6o1g.oqts.j9104768.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-fdh6f5"
Unexpected error:
    <*fmt.wrapError | 0xc000fe4500>: 
    DNS for route host agnhost-e2e-sample-app-c6klm.apps.aro.z4z2f7u9q3y4c5l.96pg.j3581568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-c6klm.apps.aro.z4z2f7u9q3y4c5l.96pg.j3581568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-c6klm.apps.aro.z4z2f7u9q3y4c5l.96pg.j3581568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-fdh6f5"
Unexpected error:
    <*fmt.wrapError | 0xc000fe4500>: 
    DNS for route host agnhost-e2e-sample-app-c6klm.apps.aro.z4z2f7u9q3y4c5l.96pg.j3581568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-c6klm.apps.aro.z4z2f7u9q3y4c5l.96pg.j3581568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-c6klm.apps.aro.z4z2f7u9q3y4c5l.96pg.j3581568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-8gb9mx"
Unexpected error:
    <*fmt.wrapError | 0xc0011931e0>: 
    DNS for route host agnhost-e2e-sample-app-28d65.apps.aro.o4p4d0n2f5l6i5t.cztj.j0999424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-28d65.apps.aro.o4p4d0n2f5l6i5t.cztj.j0999424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-28d65.apps.aro.o4p4d0n2f5l6i5t.cztj.j0999424.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-8gb9mx"
Unexpected error:
    <*fmt.wrapError | 0xc0011931e0>: 
    DNS for route host agnhost-e2e-sample-app-28d65.apps.aro.o4p4d0n2f5l6i5t.cztj.j0999424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-28d65.apps.aro.o4p4d0n2f5l6i5t.cztj.j0999424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-28d65.apps.aro.o4p4d0n2f5l6i5t.cztj.j0999424.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:45	PR #6359	2102182452419104768
2026-09-21 18:01	PR #7044	2102095923193581568
2026-09-21 13:00	PR #7033	2102020127040999424
2026-09-21 08:24	PR #6886	2101950652551794688

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00100e260>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mqf2gjsr2md2" to finish creating: GET https://rp.j9521664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b8a1157-dcd4-4bd7-b89a-f865c271214f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b8a1157-dcd4-4bd7-b89a-f865c271214f",
      "name": "5b8a1157-dcd4-4bd7-b89a-f865c271214f",
      "status": "Failed",
      "startTime": "2026-09-21T17:01:12.189816506Z",
      "endTime": "2026-09-21T17:20:16.669273518Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00100e260>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mqf2gjsr2md2" to finish creating: GET https://rp.j9521664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b8a1157-dcd4-4bd7-b89a-f865c271214f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b8a1157-dcd4-4bd7-b89a-f865c271214f",
      "name": "5b8a1157-dcd4-4bd7-b89a-f865c271214f",
      "status": "Failed",
      "startTime": "2026-09-21T17:01:12.189816506Z",
      "endTime": "2026-09-21T17:20:16.669273518Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000b37240>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-gvbn8z" in resourcegroup="rg-np-version-upgrade-gvbn8z-hkxf5kh952mz" to finish creating: GET https://rp.j9550080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34cf9dc9-964d-48aa-a9fe-30aa7b3ae5d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34cf9dc9-964d-48aa-a9fe-30aa7b3ae5d7",
      "name": "34cf9dc9-964d-48aa-a9fe-30aa7b3ae5d7",
      "status": "Failed",
      "startTime": "2026-09-21T16:16:27.808191119Z",
      "endTime": "2026-09-21T16:35:35.728389671Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000b37240>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-gvbn8z" in resourcegroup="rg-np-version-upgrade-gvbn8z-hkxf5kh952mz" to finish creating: GET https://rp.j9550080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34cf9dc9-964d-48aa-a9fe-30aa7b3ae5d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34cf9dc9-964d-48aa-a9fe-30aa7b3ae5d7",
      "name": "34cf9dc9-964d-48aa-a9fe-30aa7b3ae5d7",
      "status": "Failed",
      "startTime": "2026-09-21T16:16:27.808191119Z",
      "endTime": "2026-09-21T16:35:35.728389671Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc00019f1c0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-whfs4ngcqkr6" to finish creating: GET https://rp.j8631040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32525d6c-adbe-46db-ae4c-1e7b3b4cc233
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32525d6c-adbe-46db-ae4c-1e7b3b4cc233",
      "name": "32525d6c-adbe-46db-ae4c-1e7b3b4cc233",
      "status": "Failed",
      "startTime": "2026-09-21T02:56:41.463150178Z",
      "endTime": "2026-09-21T03:15:45.880215659Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc00019f1c0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-whfs4ngcqkr6" to finish creating: GET https://rp.j8631040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32525d6c-adbe-46db-ae4c-1e7b3b4cc233
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/32525d6c-adbe-46db-ae4c-1e7b3b4cc233",
      "name": "32525d6c-adbe-46db-ae4c-1e7b3b4cc233",
      "status": "Failed",
      "startTime": "2026-09-21T02:56:41.463150178Z",
      "endTime": "2026-09-21T03:15:45.880215659Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-21 15:45	PR #6688	2102065610849521664
2026-09-21 15:00	PR #6995	2102052987269550080
2026-09-21 02:00	PR #6965	2101853992668631040

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T14:45:59Z
  Ended:       2026-09-21T14:51:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0999424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0999424/providers/microsoft.monitor/accounts/services-j0999424", namespace="klusterlet-2svepmb9pe39k53ku2oiti573ipu6mda", pod="arohcp-monitor-kube-state-metrics-7955558448-mbr7z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2svepmb9pe39k53ku2oiti573ipu6mda/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T14:45:59Z
  Ended:       2026-09-21T14:51:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0999424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0999424/providers/microsoft.monitor/accounts/services-j0999424", namespace="klusterlet-2svepmb9pe39k53ku2oiti573ipu6mda", pod="arohcp-monitor-kube-state-metrics-7955558448-mbr7z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2svepmb9pe39k53ku2oiti573ipu6mda/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T13:19:24Z
  Ended:       2026-09-21T13:25:26Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1617792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.251:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1617792/providers/microsoft.monitor/accounts/services-j1617792", namespace="klusterlet-2svdhspilu7hjev3eenqsfgat6bij060", pod="arohcp-monitor-kube-state-metrics-7955558448-jqkl2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2svdhspilu7hjev3eenqsfgat6bij060/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T13:19:24Z
  Ended:       2026-09-21T13:25:26Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1617792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.251:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1617792/providers/microsoft.monitor/accounts/services-j1617792", namespace="klusterlet-2svdhspilu7hjev3eenqsfgat6bij060", pod="arohcp-monitor-kube-state-metrics-7955558448-jqkl2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2svdhspilu7hjev3eenqsfgat6bij060/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-21T13:19:24Z
  Ended:       2026-09-21T13:25:25Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1617792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.251:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1617792/providers/microsoft.monitor/accounts/services-j1617792", namespace="klusterlet-2svdhspilu7hjev3eenqsfgat6bij060", pod="arohcp-monitor-kube-state-metrics-7955558448-jqkl2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2svdhspilu7hjev3eenqsfgat6bij060/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-21T13:19:24Z
  Ended:       2026-09-21T13:25:25Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1617792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.251:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1617792/providers/microsoft.monitor/accounts/services-j1617792", namespace="klusterlet-2svdhspilu7hjev3eenqsfgat6bij060", pod="arohcp-monitor-kube-state-metrics-7955558448-jqkl2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2svdhspilu7hjev3eenqsfgat6bij060/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T02:53:04Z
  Ended:       2026-09-21T03:00:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0544384-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.205:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0544384/providers/microsoft.monitor/accounts/services-j0544384", namespace="klusterlet-2sv4cqnh3tg4c1kjg6jvt3g7omcg6sqq", pod="arohcp-monitor-kube-state-metrics-7955558448-hv4kl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sv4cqnh3tg4c1kjg6jvt3g7omcg6sqq/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T02:53:05Z
  Ended:       2026-09-21T02:59:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0544384-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.205:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0544384/providers/microsoft.monitor/accounts/services-j0544384", namespace="klusterlet-2sv4cmlktjpm9h9svdn985c6grh4d055", pod="arohcp-monitor-kube-state-metrics-7955558448-hv4kl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sv4cmlktjpm9h9svdn985c6grh4d055/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-21 13:00	PR #7033	2102020127040999424
2026-09-21 12:00	PR #7044	2102005289971617792
2026-09-21 01:27	PR #7000	2101845814870544384

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T00:51:04Z
  Ended:       2026-09-22T01:02:04Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9104768-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T07:46:48Z
  Ended:       2026-09-21T07:52:47Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4549760-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T04:18:24Z
  Ended:       2026-09-21T04:26:26Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3757440-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:45	PR #6359	2102182452419104768
2026-09-21 06:45	PR #7042	2101925644194549760
2026-09-21 03:02	PR #6225	2101869717533757440

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (3)

time=2026-09-21T18:34:08.196Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T18:41:45.259Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T18:34:08.202514223Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-21T18:34:09.977044373Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T18:34:10.113714041Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-21T18:41:45.256724505Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"63BD079BDF59F7AF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j5917696-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:34:14.3081847Z\",\"duration\":\"PT7M21.1354492S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:41:45.256785637Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"66DC8ACA57717BD0\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j5917696-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:39:37.4650836Z\",\"duration\":\"PT4.3067939S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:41:45.256818608Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A4AD211D64497EC2\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j5917696-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:39:37.4650836Z\",\"duration\":\"PT1M56.945679S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:41:45.256835428Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"FAD9AF8022EC4353\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j5917696-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:39:37.4650836Z\",\"duration\":\"PT3.9630448S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:41:45.256866349Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E51E80C79F3CC641\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j5917696-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:39:37.4650836Z\",\"duration\":\"PT3.4005374S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:41:45.25689421Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"640F2A04A52AF91E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1\",\"resourceName\":\"ci01-j5917696-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:34:14.3081847Z\",\"duration\":\"PT5M23.0886527S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:41:45.25691446Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5917696-mgmt-1\",\"clusterName\":\"ci01-j5917696-mgmt-1\",\"region\":\"canadacentral\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115912350974327\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5917696-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5917696-mgmt-1/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m37.054409927s\"}\n exit status 1"

time=2026-09-21T18:33:10.189Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T18:43:18.824Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T18:33:10.196040039Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-21T18:33:12.013713201Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T18:33:12.155143709Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-21T18:43:18.82214017Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2B38A36D9799EB2E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j0180608-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:33:16.7353105Z\",\"duration\":\"PT9M44.0435529S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:43:18.822198134Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"D52FC66163407372\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j0180608-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:41:21.3624777Z\",\"duration\":\"PT1M41.8443951S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:43:18.822211544Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"8650A5AE123230FA\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j0180608-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:41:21.3624777Z\",\"duration\":\"PT4.1420203S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:43:18.822237446Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"7AC95EDE577F9BF6\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j0180608-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:41:21.3624777Z\",\"duration\":\"PT4.0795266S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:43:18.822259057Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"7610750267384434\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j0180608-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:41:21.3624777Z\",\"duration\":\"PT3.4857707S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:43:18.822279099Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"CCE749298B22E92C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1\",\"resourceName\":\"ci01-j0180608-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:33:16.7353105Z\",\"duration\":\"PT8M4.5504301S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:43:18.82230049Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0180608-mgmt-1\",\"clusterName\":\"ci01-j0180608-mgmt-1\",\"region\":\"canadacentral\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115912930590660\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0180608-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0180608-mgmt-1/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"10m8.626268442s\"}\n exit status 1"

time=2026-09-21T18:32:22.796Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-21T18:45:32.382Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-21T18:32:22.803573824Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-21T18:32:24.805955581Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-21T18:32:24.940582858Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-21T18:45:32.379977629Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"EBBBD7B642ED7B1C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j9776896-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:43:20.9144849Z\",\"duration\":\"PT4.4919188S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:45:32.380066075Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"379E505D806B0369\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j9776896-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:32:30.2828584Z\",\"duration\":\"PT12M29.3745304S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:45:32.380090146Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"5262EBE64FE1B417\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j9776896-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:43:20.9144849Z\",\"duration\":\"PT3.9450427S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:45:32.380121888Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"ED59D84AC1FC6A35\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j9776896-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-21T18:43:20.9144849Z\",\"duration\":\"PT3.7887958S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-21T18:45:32.38015029Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"4C2369403F5BC6DA\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j9776896-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:43:20.9144849Z\",\"duration\":\"PT1M43.0824083S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:45:32.3801626Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"029C5991DAA3D2F1\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1\",\"resourceName\":\"ci01-j9776896-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-21T18:32:30.2828584Z\",\"duration\":\"PT10M50.5766775S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-21T18:45:32.380190652Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9776896-mgmt-1\",\"clusterName\":\"ci01-j9776896-mgmt-1\",\"region\":\"canadacentral\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115913402802699\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9776896-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j9776896-mgmt-1/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 6 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"13m9.576625468s\"}\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-21 18:12	PR #7053	2102098571925917696
2026-09-21 18:05	PR #5562	2102096991210180608
2026-09-21 18:01	PR #6995	2102095918189776896

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (2)

alert fired 23 time(s) (3 unknown, 20 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T04:06:07Z
  Ended:       2026-09-21T04:14:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9314560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv5p6n0mkeb9fggti8c47tsk9keuq7l-agg-cluster", pod="etcd-2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sv5p6n0mkeb9fggti8c47tsk9keuq7l-agg-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T04:07:06Z
  Ended:       2026-09-21T04:14:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9314560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster", pod="etcd-2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-21T04:07:06Z
  Ended:       2026-09-21T04:14:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9314560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster", pod="ignition-server-proxy-6ccf8c6444-24qfp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sv5ortu9h2o8a77c86l3q2dojqo18s5-cilium-cluster/ignition-server-proxy-6ccf8c6444-24qfp has been in a non-ready state for longer than 5 minutes.

alert fired 25 time(s) (2 unknown, 23 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T03:29:18Z
  Ended:       2026-09-21T03:35:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6553344-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv56oltrdjme8j8laq16dpf3uvs62o8-x7r1k4p1e9k2r7a", pod="cluster-network-operator-6c8887dd6f-5lxrp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2sv56oltrdjme8j8laq16dpf3uvs62o8-x7r1k4p1e9k2r7a/cluster-network-operator-6c8887dd6f-5lxrp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-21T03:29:18Z
  Ended:       2026-09-21T03:35:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6553344-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sv56oltrdjme8j8laq16dpf3uvs62o8-x7r1k4p1e9k2r7a", pod="ingress-operator-8459995bdf-h2f6l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2sv56oltrdjme8j8laq16dpf3uvs62o8-x7r1k4p1e9k2r7a/ingress-operator-8459995bdf-h2f6l has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-21 03:02	PR #7032	2101869655269314560
2026-09-21 02:16	PR #7036	2101858124586553344

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T21:03:18Z
  Ended:       2026-09-21T21:18:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5527296-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5527296/providers/microsoft.monitor/accounts/hcps-j5527296", namespace="ocm-arohcpci01-2svk9tvinsjpvsqf4f95mmif44fb0pv7-v4j2g0l9y3n4g2w", pod="arohcp-monitor-kube-state-metrics-7955558448-qjq7d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2svk9tvinsjpvsqf4f95mmif44fb0pv7-v4j2g0l9y3n4g2w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T14:46:09Z
  Ended:       2026-09-21T14:58:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0999424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0999424/providers/microsoft.monitor/accounts/hcps-j0999424", namespace="ocm-arohcpci01-2svepmb9pe39k53ku2oiti573ipu6mda-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7955558448-mbr7z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2svepmb9pe39k53ku2oiti573ipu6mda-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-21 19:38	PR #6925	2102120305865527296
2026-09-21 13:00	PR #7033	2102020127040999424

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T19:14:35Z
  Ended:       2026-09-21T19:47:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3834368-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3834368/providers/microsoft.monitor/accounts/hcps-j3834368", namespace="ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e050989a-8cff-4b23-a05f-7aa2cc9bd30a"
  Description: Pod ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-21T19:14:35Z
  Ended:       2026-09-21T19:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3834368-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3834368/providers/microsoft.monitor/accounts/hcps-j3834368", namespace="ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e050989a-8cff-4b23-a05f-7aa2cc9bd30a"
  Description: Pod ocm-arohcpci01-2sviu319g5e6he82ppjsp4nds4cimm8g-o7o2u6m5s2o5a7f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T13:07:01Z
  Ended:       2026-09-21T13:34:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1617792-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.251:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1617792/providers/microsoft.monitor/accounts/hcps-j1617792", namespace="ocm-arohcpci01-2svdhspilu7hjev3eenqsfgat6bij060-t5x1q0q6n3v3g4d", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8d9afbb2-38d4-42d1-9c00-d0aab022a452"
  Description: Pod ocm-arohcpci01-2svdhspilu7hjev3eenqsfgat6bij060-t5x1q0q6n3v3g4d/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-21T13:08:02Z
  Ended:       2026-09-21T13:37:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1617792-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.251:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1617792/providers/microsoft.monitor/accounts/hcps-j1617792", namespace="ocm-arohcpci01-2svdhspilu7hjev3eenqsfgat6bij060-t5x1q0q6n3v3g4d", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8d9afbb2-38d4-42d1-9c00-d0aab022a452"
  Description: Pod ocm-arohcpci01-2svdhspilu7hjev3eenqsfgat6bij060-t5x1q0q6n3v3g4d/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-21 18:01	PR #6886	2102095915643834368
2026-09-21 12:00	PR #7044	2102005289971617792

client rate limiter Wait returned an error: context deadline exceeded

full failure pattern:

client rate limiter Wait returned an error: context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002a16b28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002a16b28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0010c2048>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-ng5tlp namespace are ready: failed to list pods in cilium-connectivity-check-ng5tlp namespace: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0010c2048>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-ng5tlp namespace are ready: failed to list pods in cilium-connectivity-check-ng5tlp namespace: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-21 18:01	PR #7033	2102095920672804864
2026-09-21 02:16	PR #7036	2101858124586553344

alert [hcp] userJourneySwiftRouterAvailability1h5m fired

full failure pattern:

alert [hcp] userJourneySwiftRouterAvailability1h5m fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T00:47:01Z
  Ended:       2026-09-22T01:21:03Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j9104768-mgmt-1", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9104768/providers/microsoft.monitor/accounts/hcps-j9104768", region="westus3", severity="3", short_window="5m", slo="swift-router-availability"
  Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-21T17:49:18Z
  Ended:       2026-09-21T17:59:19Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j9215744-mgmt-2", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9215744/providers/microsoft.monitor/accounts/hcps-j9215744", region="westus3", severity="3", short_window="5m"
  Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneySwiftRouterAvailability1h5m does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:45	PR #6359	2102182452419104768
2026-09-21 16:50	PR #6359	2102079067309215744

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000944ce0>: 
    failed to create HCP cluster np-downgrade-pg2f5g: failed waiting for cluster="np-downgrade-pg2f5g" in resourcegroup="rg-np-downgrade-pg2f5g-x9t2rw964n4f" to finish creating: GET https://rp.j0544384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4eb55b38-0373-4fce-befc-4d5851c7acc1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4eb55b38-0373-4fce-befc-4d5851c7acc1",
      "name": "4eb55b38-0373-4fce-befc-4d5851c7acc1",
      "status": "Failed",
      "startTime": "2026-09-21T02:20:57.450123548Z",
      "endTime": "2026-09-21T02:50:04.742128516Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000944ce0>: 
    failed to create HCP cluster np-downgrade-pg2f5g: failed waiting for cluster="np-downgrade-pg2f5g" in resourcegroup="rg-np-downgrade-pg2f5g-x9t2rw964n4f" to finish creating: GET https://rp.j0544384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4eb55b38-0373-4fce-befc-4d5851c7acc1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4eb55b38-0373-4fce-befc-4d5851c7acc1",
      "name": "4eb55b38-0373-4fce-befc-4d5851c7acc1",
      "status": "Failed",
      "startTime": "2026-09-21T02:20:57.450123548Z",
      "endTime": "2026-09-21T02:50:04.742128516Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000fe62e0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-b7gtsbk76cn9" to finish creating: GET https://rp.j0544384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/627ebc1b-6227-4297-a3c2-d1736e67605d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/627ebc1b-6227-4297-a3c2-d1736e67605d",
      "name": "627ebc1b-6227-4297-a3c2-d1736e67605d",
      "status": "Failed",
      "startTime": "2026-09-21T02:21:40.675582383Z",
      "endTime": "2026-09-21T02:50:44.756111292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000fe62e0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-b7gtsbk76cn9" to finish creating: GET https://rp.j0544384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/627ebc1b-6227-4297-a3c2-d1736e67605d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/627ebc1b-6227-4297-a3c2-d1736e67605d",
      "name": "627ebc1b-6227-4297-a3c2-d1736e67605d",
      "status": "Failed",
      "startTime": "2026-09-21T02:21:40.675582383Z",
      "endTime": "2026-09-21T02:50:44.756111292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000a64c80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-lzbpqchwtpjq" to finish creating: GET https://rp.j0544384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/50782cfe-84f8-43a4-836f-7dfb3ca52030
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/50782cfe-84f8-43a4-836f-7dfb3ca52030",
      "name": "50782cfe-84f8-43a4-836f-7dfb3ca52030",
      "status": "Failed",
      "startTime": "2026-09-21T02:21:02.502165407Z",
      "endTime": "2026-09-21T02:50:05.126194832Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000a64c80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-lzbpqchwtpjq" to finish creating: GET https://rp.j0544384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/50782cfe-84f8-43a4-836f-7dfb3ca52030
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/50782cfe-84f8-43a4-836f-7dfb3ca52030",
      "name": "50782cfe-84f8-43a4-836f-7dfb3ca52030",
      "status": "Failed",
      "startTime": "2026-09-21T02:21:02.502165407Z",
      "endTime": "2026-09-21T02:50:05.126194832Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (29)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 01:27	PR #7000	2101845814870544384

clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>

full failure pattern:

clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc0023f6980>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:51502->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc0023f6980>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:51502->172.30.0.10:53: read: connection refused
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 08:24	PR #6886	2101950652551794688

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

full failure pattern:

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 01:27	PR #7000	2101845814870544384

at least one schedule should have a LastBackupTime

full failure pattern:

at least one schedule should have a LastBackupTime

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.000s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.000s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE can pause schedules to stop backup execution for an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 11:46	PR #6965	2102001525193707520

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [openshift-apiserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b94ca0>: 
    failed to create HCP cluster np-downgrade-78fkhm: failed waiting for cluster="np-downgrade-78fkhm" in resourcegroup="rg-np-downgrade-78fkhm-mnlqzpfzdmbk" to finish creating: GET https://rp.j9591552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ed3e6b5-4161-4349-9abf-4a92c51749bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ed3e6b5-4161-4349-9abf-4a92c51749bd",
      "name": "4ed3e6b5-4161-4349-9abf-4a92c51749bd",
      "status": "Failed",
      "startTime": "2026-09-21T09:54:06.067313752Z",
      "endTime": "2026-09-21T10:23:08.770391006Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 7s ago; version 4.21.34 is Partial (want Completed), started 5m11s ago; version 4.21.34 is Partial (want Completed), started 10m32s ago; version 4.21.34 is Partial (want Completed), started 14m36s ago; version 4.21.34 is Partial (want Completed), started 19m54s ago; version 4.21.34 is Partial (want Completed), started 23m29s ago; version 4.21.34 is Partial (want Completed), started 24m30s ago; version 4.21.34 is Partial (want Completed), started 25m7s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b94ca0>: 
    failed to create HCP cluster np-downgrade-78fkhm: failed waiting for cluster="np-downgrade-78fkhm" in resourcegroup="rg-np-downgrade-78fkhm-mnlqzpfzdmbk" to finish creating: GET https://rp.j9591552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ed3e6b5-4161-4349-9abf-4a92c51749bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ed3e6b5-4161-4349-9abf-4a92c51749bd",
      "name": "4ed3e6b5-4161-4349-9abf-4a92c51749bd",
      "status": "Failed",
      "startTime": "2026-09-21T09:54:06.067313752Z",
      "endTime": "2026-09-21T10:23:08.770391006Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 7s ago; version 4.21.34 is Partial (want Completed), started 5m11s ago; version 4.21.34 is Partial (want Completed), started 10m32s ago; version 4.21.34 is Partial (want Completed), started 14m36s ago; version 4.21.34 is Partial (want Completed), started 19m54s ago; version 4.21.34 is Partial (want Completed), started 23m29s ago; version 4.21.34 is Partial (want Completed), started 24m30s ago; version 4.21.34 is Partial (want Completed), started 25m7s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 09:02	PR #6995	2101960313149591552

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e3ab10>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-jcvfb4wmbljw" to finish deleting: GET https://rp.j9663104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fdbca9d2-e9b2-491a-8405-4b0598064c63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fdbca9d2-e9b2-491a-8405-4b0598064c63",
      "name": "fdbca9d2-e9b2-491a-8405-4b0598064c63",
      "status": "Failed",
      "startTime": "2026-09-21T10:31:21.582663948Z",
      "endTime": "2026-09-21T10:55:21.85126634Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svb4o21n7e1d1vgdhv23sdiarqs1kr9 still exists (deletion dispatched at 2026-09-21T10:31:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e3ab10>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-jcvfb4wmbljw" to finish deleting: GET https://rp.j9663104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fdbca9d2-e9b2-491a-8405-4b0598064c63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fdbca9d2-e9b2-491a-8405-4b0598064c63",
      "name": "fdbca9d2-e9b2-491a-8405-4b0598064c63",
      "status": "Failed",
      "startTime": "2026-09-21T10:31:21.582663948Z",
      "endTime": "2026-09-21T10:55:21.85126634Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svb4o21n7e1d1vgdhv23sdiarqs1kr9 still exists (deletion dispatched at 2026-09-21T10:31:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 09:24	PR #7044	2101965713999663104

UpdateHCPCluster20260901 timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

UpdateHCPCluster20260901 timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:184]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015dccf0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-fm6rvv7nj2n4" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-fm6rvv7nj2n4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:184]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015dccf0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-fm6rvv7nj2n4" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-fm6rvv7nj2n4, error: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 21:00	PR #7044	2102140947914559488

alert [svc] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [svc] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-21T02:59:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0544384-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.205:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0544384/providers/microsoft.monitor/accounts/services-j0544384", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-hv4kl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-21T02:59:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0544384-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.205:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0544384/providers/microsoft.monitor/accounts/services-j0544384", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-hv4kl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 01:27	PR #7000	2101845814870544384

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-21T02:21:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0544384-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-54f7d49479-4gspp", region="centralus", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f7d49479-4gspp has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 01:27	PR #7000	2101845814870544384

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-pkrdhh"
Unexpected error:
    <*fmt.wrapError | 0xc0005f0960>: 
    DNS for route host agnhost-e2e-sample-app-zhfc5.apps.aro.e2n6l9t1x7s8m0x.srbh.j1324544.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zhfc5.apps.aro.e2n6l9t1x7s8m0x.srbh.j1324544.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zhfc5.apps.aro.e2n6l9t1x7s8m0x.srbh.j1324544.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-pkrdhh"
Unexpected error:
    <*fmt.wrapError | 0xc0005f0960>: 
    DNS for route host agnhost-e2e-sample-app-zhfc5.apps.aro.e2n6l9t1x7s8m0x.srbh.j1324544.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zhfc5.apps.aro.e2n6l9t1x7s8m0x.srbh.j1324544.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zhfc5.apps.aro.e2n6l9t1x7s8m0x.srbh.j1324544.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 15:22	PR #6891	2102071948841324544

ERROR CODE: DeploymentFailed; detail code ConflictError; detail message A conflict occurred that prevented the operation from completing. The operation failed because the Key Vaul...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ConflictError; detail message A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: <url>

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T08:59:12.017Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra description="Step infra\n  Kind: ARM\n  Template: templates/svc-infra.bicep\n  Parameters: configurations/svc-infra.tmpl.bicepparam"
time=2026-09-21T08:59:13.543Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra
time=2026-09-21T08:59:14.650Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra deployment=2380d2dc5c272e55d92eda779d891cdc0c622a661aa6f1f76f2e7d85c1855d68 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4094720-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2380d2dc5c272e55d92eda779d891cdc0c622a661aa6f1f76f2e7d85c1855d68
time=2026-09-21T08:59:44.965Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4094720-svc/providers/Microsoft.Resources/deployments/2380d2dc5c272e55d92eda779d891cdc0c622a661aa6f1f76f2e7d85c1855d68/operationStatuses/08584116257315009198\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/svc-kv-6eqsxh5wya5uu\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ConflictError\\\",\\r\\n            \\\"message\\\": \\\"A conflict occurred that prevented the operation from completing. The operation failed because the Key Vault '' changed from the point the operation began. This can happen if parallel operations are being performed on the Key Vault. To prevent this error, serialize the operations so that only one operation is performed on the Key Vault at a time. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2147741\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/infra	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 08:04	PR #6948	2101945646264094720

ERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Resources/deployments/svc-kv-xehipxbjxawki' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '<timestamp>' with correlationId '<uuid>', and will expire at '<timestamp>' if it does not complete before then.

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T09:34:53.623Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output description="Step infra-output\n  Kind: ARM\n  Template: templates/svc-infra-lookup.bicep\n  Parameters: configurations/svc-infra-lookup.tmpl.bicepparam"
time=2026-09-21T09:34:54.045Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output
time=2026-09-21T09:34:54.556Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output deployment=4b9be9d2fe9447c77bb0d4b5c4352163b3563c2d648c6aabae0bd3896f58b810 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8047872-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4b9be9d2fe9447c77bb0d4b5c4352163b3563c2d648c6aabae0bd3896f58b810
time=2026-09-21T09:35:24.831Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8047872-svc/providers/Microsoft.Resources/deployments/4b9be9d2fe9447c77bb0d4b5c4352163b3563c2d648c6aabae0bd3896f58b810/operationStatuses/08584116235912314952\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"DeploymentActive\\\",\\r\\n    \\\"message\\\": \\\"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/svc-kv-xehipxbjxawki' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '9/21/2026 9:34:50 AM' with correlationId 'f4ad37b9-8244-4bad-ac82-9eb55023bc2b', and will expire at '9/28/2026 9:34:50 AM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/infra-output	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 09:00	PR #7033	2101959670708047872

ERROR CODE: DeploymentFailed; detail code DeploymentNotFound; detail message Deployment 'svc-kv-xehipxbjxawki' could not be found.

full failure pattern:

ERROR CODE: DeploymentFailed; detail code DeploymentNotFound; detail message Deployment 'svc-kv-xehipxbjxawki' could not be found.

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T12:09:00.128Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output description="Step infra-output\n  Kind: ARM\n  Template: templates/svc-infra-lookup.bicep\n  Parameters: configurations/svc-infra-lookup.tmpl.bicepparam"
time=2026-09-21T12:09:00.546Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output
time=2026-09-21T12:09:01.165Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output deployment=44b6ca1d6f0df81628d0ba835badac62186b733c716b254d3b8b1c680dfe43f9 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6430592-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F44b6ca1d6f0df81628d0ba835badac62186b733c716b254d3b8b1c680dfe43f9
time=2026-09-21T12:09:31.310Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6430592-svc/providers/Microsoft.Resources/deployments/44b6ca1d6f0df81628d0ba835badac62186b733c716b254d3b8b1c680dfe43f9/operationStatuses/08584116143446905098\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"NotFound\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"DeploymentNotFound\\\",\\r\\n    \\\"message\\\": \\\"Deployment 'svc-kv-xehipxbjxawki' could not be found.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/infra-output	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 12:00	PR #6948	2102005345336430592

ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:85]: failed to create customer resources for external auth cluster
Unexpected error:
    <*fmt.wrapError | 0xc00072ec40>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-sntc52" in resourcegroup="external-auth-cluster-bkt2n5ldrfvt" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-bkt2n5ldrfvt/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-sntc52/operationStatuses/08584116072201723846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xkv3iaodnbzd2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:85]: failed to create customer resources for external auth cluster
Unexpected error:
    <*fmt.wrapError | 0xc00072ec40>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-sntc52" in resourcegroup="external-auth-cluster-bkt2n5ldrfvt" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-bkt2n5ldrfvt/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-sntc52/operationStatuses/08584116072201723846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xkv3iaodnbzd2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 13:00	PR #7033	2102020127040999424

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: mu...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c2c4c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-z7kkssjppzxn" to finish creating: GET https://rp.j9110912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b42d1d15-2b65-453b-bd99-8dc808b1a981
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b42d1d15-2b65-453b-bd99-8dc808b1a981",
      "name": "b42d1d15-2b65-453b-bd99-8dc808b1a981",
      "status": "Failed",
      "startTime": "2026-09-21T09:25:09.210146348Z",
      "endTime": "2026-09-21T09:54:10.812258416Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, catalog-operator, redhat-marketplace-catalog, hosted-cluster-config-operator, ingress-operator, olm-operator, redhat-operators-catalog, dns-operator, community-operators-catalog, csi-snapshot-controller-operator, certified-operators-catalog, cluster-version-operator, cluster-network-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c2c4c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-z7kkssjppzxn" to finish creating: GET https://rp.j9110912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b42d1d15-2b65-453b-bd99-8dc808b1a981
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b42d1d15-2b65-453b-bd99-8dc808b1a981",
      "name": "b42d1d15-2b65-453b-bd99-8dc808b1a981",
      "status": "Failed",
      "startTime": "2026-09-21T09:25:09.210146348Z",
      "endTime": "2026-09-21T09:54:10.812258416Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, catalog-operator, redhat-marketplace-catalog, hosted-cluster-config-operator, ingress-operator, olm-operator, redhat-operators-catalog, dns-operator, community-operators-catalog, csi-snapshot-controller-operator, certified-operators-catalog, cluster-version-operator, cluster-network-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE can pause schedules to stop backup execution for an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 08:39	PR #5724	2101954553699110912

ERROR CODE: ServerTimeout

full failure pattern:

ERROR CODE: ServerTimeout

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T19:22:42.604Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet-permissions stamp=2 description="Step swift-vnet-permissions\n  Kind: ARM\n  Template: templates/swift-vnet-permissions.bicep\n  Parameters: configurations/swift-vnet-permissions.tmpl.bicepparam"
time=2026-09-21T19:23:35.587Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet-permissions stamp=2
time=2026-09-21T19:25:31.977Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet-permissions stamp=2 err="stamp 2: failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4252800-mgmt-2/providers/Microsoft.Resources/deployments/681f37eea538402bd4d4d12ef523e5767a0f365997d5c7810d0e142ad8086026\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServerTimeout\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServerTimeout\",\n    \"message\": \"The request timed out. Diagnostic information: timestamp '20260921T192531Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0d759d1e-a59a-434f-931d-09ba48ef8f44', request correlation id '0d759d1e-a59a-434f-931d-09ba48ef8f44'.\"\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/swift-vnet-permissions (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 18:52	PR #6945	2102108631234252800

alert [svc] KustoLogsDataStale fired

full failure pattern:

alert [svc] KustoLogsDataStale fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-22T00:14:32Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0434432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.56:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0434432/providers/microsoft.monitor/accounts/services-j0434432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-9fd595fb9-74n8n", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0434432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-22T00:14:32Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0434432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.56:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0434432/providers/microsoft.monitor/accounts/services-j0434432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-9fd595fb9-74n8n", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0434432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 22:11	PR #7057	2102158779100434432

error: You must be logged in to the server (the server has asked for the client to provide credentials)

full failure pattern:

error: You must be logged in to the server (the server has asked for the client to provide credentials)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 305.584s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc001028de0>: 
    E0921 23:06:57.092173    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.110667    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.129290    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.149503    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.163374    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 305.584s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc001028de0>: 
    E0921 23:06:57.092173    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.110667    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.129290    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.149503    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    E0921 23:06:57.163374    2855 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 21:51	PR #6945	2102153867507535872

failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress context deadline exceed...

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-09-21T17:32:18.785Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-09-21T17:32:19.291Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23]"
time=2026-09-21T17:32:19.291Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-21T17:32:19.604Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-09-21T17:32:19.952Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-09-21T17:32:19.952Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-21T17:32:20.082Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-21T17:37:24.746Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-09-21T17:37:24.746Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/storageclass (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 14:23	PR #7044	2102079066994642944

alert [hcp] userJourneySwiftRouterAvailability6h30m fired

full failure pattern:

alert [hcp] userJourneySwiftRouterAvailability6h30m fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-22T01:26:03Z
  Ended:       2026-09-22T01:32:03Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability6h30m", burn_rate_tier="medium", cluster="ci01-j9104768-mgmt-2", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9104768/providers/microsoft.monitor/accounts/hcps-j9104768", region="westus3", severity="3", short_window="30m", slo="swift-router-availability"
  Description: SWIFT router-capacity availability is consuming its error budget at the medium burn rate.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneySwiftRouterAvailability6h30m does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:45	PR #6359	2102182452419104768

alert [svc] BackendControllerRetryHotLoop fired

full failure pattern:

alert [svc] BackendControllerRetryHotLoop fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-21T17:00:35Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j4484992-svc", component="backend", name="controlplaneversionstatuscollector", region="westus3", severity="warning"
  Description: Backend controller workqueue controlplaneversionstatuscollector has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 15:43	PR #6577	2102066961654484992

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00125b6a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5248mpsj56q6" to finish creating: GET https://rp.j9521664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd0358b-031c-46ab-9abd-cd357255bda0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd0358b-031c-46ab-9abd-cd357255bda0",
      "name": "0cd0358b-031c-46ab-9abd-cd357255bda0",
      "status": "Failed",
      "startTime": "2026-09-21T17:03:39.705515335Z",
      "endTime": "2026-09-21T17:22:46.542422976Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00125b6a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5248mpsj56q6" to finish creating: GET https://rp.j9521664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd0358b-031c-46ab-9abd-cd357255bda0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd0358b-031c-46ab-9abd-cd357255bda0",
      "name": "0cd0358b-031c-46ab-9abd-cd357255bda0",
      "status": "Failed",
      "startTime": "2026-09-21T17:03:39.705515335Z",
      "endTime": "2026-09-21T17:22:46.542422976Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 15:45	PR #6688	2102065610849521664

kube_node_info metrics never appeared in Azure Monitor for any hostedcontrolplane

full failure pattern:

kube_node_info metrics never appeared in Azure Monitor for any hostedcontrolplane

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:91]: Timed out after 900.001s.
kube_node_info metrics never appeared in Azure Monitor for any hostedcontrolplane
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:89 with:
expected kube_node_info metrics for at least one hostedcontrolplane but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:91]: Timed out after 900.001s.
kube_node_info metrics never appeared in Azure Monitor for any hostedcontrolplane
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:89 with:
expected kube_node_info metrics for at least one hostedcontrolplane but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	KSM HCP Metrics kube_node_info metrics should be present in Azure Monitor for the happy-path cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 13:20	PR #7009	2102025100818649088

Environment: INT

Failure patterns: 1 · Failures matched: 1 · Total runs: 5 · Runs affected: 1

Failure pattern

Failed at

Trend

Also in

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

full failure pattern:

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000b78000>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000b78000>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 07:58	n/a	2101944127254958080

Environment: PROD

Failure patterns: 6 · Failures matched: 13 · Total runs: 2 · Runs affected: 2

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc002e0bcb0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-5nj7xwd9d8lp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc002e0bcb0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-5nj7xwd9d8lp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a388d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-hnrk7b4v2x6b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a388d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-hnrk7b4v2x6b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0028a88a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rng5tfgqsvfr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0028a88a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rng5tfgqsvfr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:00	n/a	2102171179442769920

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000c586c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-5ldnts" in resourcegroup="rg-np-skip-minor-5ldnts-pnzgxgcmlk5s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-5ldnts-pnzgxgcmlk5s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-5ldnts/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000c586c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-5ldnts" in resourcegroup="rg-np-skip-minor-5ldnts-pnzgxgcmlk5s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-5ldnts-pnzgxgcmlk5s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-5ldnts/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000e87aa0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-jppjzb" resourcegroup="rg-np-dg-minor-jppjzb-mqbgnthqv6nw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-jppjzb-mqbgnthqv6nw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-jppjzb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000e87aa0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-jppjzb" resourcegroup="rg-np-dg-minor-jppjzb-mqbgnthqv6nw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-jppjzb-mqbgnthqv6nw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-jppjzb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-wlqfz9" for cluster "cluster-nightly-4-20-wlqfz9"
Unexpected error:
    <*fmt.wrapError | 0xc00096e3a0>: 
    failed to create NodePool np-wlqfz9: failed starting nodepool creation "np-wlqfz9" for cluster "cluster-nightly-4-20-wlqfz9" in resourcegroup="rg-nightly-4-20-7q72mxd46fc9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-7q72mxd46fc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-wlqfz9/nodePools/np-wlqfz9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-wlqfz9" for cluster "cluster-nightly-4-20-wlqfz9"
Unexpected error:
    <*fmt.wrapError | 0xc00096e3a0>: 
    failed to create NodePool np-wlqfz9: failed starting nodepool creation "np-wlqfz9" for cluster "cluster-nightly-4-20-wlqfz9" in resourcegroup="rg-nightly-4-20-7q72mxd46fc9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-7q72mxd46fc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-wlqfz9/nodePools/np-wlqfz9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:00	n/a	2102171179442769920

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc00155c2e0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-tnpj8h" in resourcegroup="rg-np-version-upgrade-tnpj8h-m6skjqh8vcv8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/670619ae-29a5-476c-912b-af06c657c780
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/670619ae-29a5-476c-912b-af06c657c780",
      "name": "670619ae-29a5-476c-912b-af06c657c780",
      "status": "Failed",
      "startTime": "2026-09-21T03:17:13.234002267Z",
      "endTime": "2026-09-21T03:36:17.606656785Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc00155c2e0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-tnpj8h" in resourcegroup="rg-np-version-upgrade-tnpj8h-m6skjqh8vcv8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/670619ae-29a5-476c-912b-af06c657c780
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/670619ae-29a5-476c-912b-af06c657c780",
      "name": "670619ae-29a5-476c-912b-af06c657c780",
      "status": "Failed",
      "startTime": "2026-09-21T03:17:13.234002267Z",
      "endTime": "2026-09-21T03:36:17.606656785Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 02:00	n/a	2101854085815734272

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00062e820>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-rnt89sjcb8mz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-rnt89sjcb8mz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00062e820>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-rnt89sjcb8mz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-rnt89sjcb8mz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:00	n/a	2102171179442769920

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0009249c0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-111728, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0009249c0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-111728, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:00	n/a	2102171179442769920

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-9dm6vxblqqwm/cluster-nightly-5-1-2mzlb4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e2a620>: 
    failed to create HCP cluster cluster-nightly-5-1-2mzlb4: failed waiting for cluster="cluster-nightly-5-1-2mzlb4" in resourcegroup="rg-nightly-5-1-9dm6vxblqqwm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5936c3-5c07-4597-b323-aa8c4ac28e70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5936c3-5c07-4597-b323-aa8c4ac28e70",
      "name": "1e5936c3-5c07-4597-b323-aa8c4ac28e70",
      "status": "Failed",
      "startTime": "2026-09-21T23:36:42.177277415Z",
      "endTime": "2026-09-22T00:05:42.40340744Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-9dm6vxblqqwm/cluster-nightly-5-1-2mzlb4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e2a620>: 
    failed to create HCP cluster cluster-nightly-5-1-2mzlb4: failed waiting for cluster="cluster-nightly-5-1-2mzlb4" in resourcegroup="rg-nightly-5-1-9dm6vxblqqwm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5936c3-5c07-4597-b323-aa8c4ac28e70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5936c3-5c07-4597-b323-aa8c4ac28e70",
      "name": "1e5936c3-5c07-4597-b323-aa8c4ac28e70",
      "status": "Failed",
      "startTime": "2026-09-21T23:36:42.177277415Z",
      "endTime": "2026-09-22T00:05:42.40340744Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:00	n/a	2102171179442769920

Environment: STG

Failure patterns: 1 · Failures matched: 108 · Total runs: 2 · Runs affected: 2

Failure pattern

Failed at

Trend

Also in

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHat...

full failure pattern:

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e60380>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-xkmrf4xtl8st": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-xkmrf4xtl8st/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e60380>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-xkmrf4xtl8st": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-xkmrf4xtl8st/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fb0120>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-g886g4ln75lz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-g886g4ln75lz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fb0120>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-g886g4ln75lz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-g886g4ln75lz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0016ea260>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-7k6gmjrrhndn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-7k6gmjrrhndn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0016ea260>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-7k6gmjrrhndn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-7k6gmjrrhndn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (108)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to delete an HCP cluster whose managed identities were deleted first	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to list available HCP OpenShift versions and validate response content	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with back-level version 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create node pools with ARM64-based VMs	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to delete a nodepool	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to delete an HCP cluster whose managed identities were deleted first	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to list available HCP OpenShift versions and validate response content	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-21 23:00	n/a	2102171179497295872
2026-09-21 02:00	n/a	2101854085849288704