CIHealth Failure Patterns

Environment: DEV

Failure patterns: 52 · Failures matched: 201 · Total runs: 123 · Runs affected: 91

Failure pattern

Failed at

Trend

Also in

not all KSM metrics appeared in Azure Monitor

full failure pattern:

not all KSM metrics appeared in Azure Monitor

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:185]: Timed out after 1500.005s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:183 with:
expected kube_node_status_condition from kube-state-metrics; kube_node_info from kube-state-metrics but got no results
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:185]: Timed out after 1500.005s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:183 with:
expected kube_node_status_condition from kube-state-metrics; kube_node_info from kube-state-metrics but got no results
Expected
    <[]string | len:2, cap:2>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:185]: Timed out after 1500.000s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:183 with:
expected kube_node_status_condition from kube-state-metrics; kube_node_info from kube-state-metrics but got no results
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:185]: Timed out after 1500.000s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:183 with:
expected kube_node_status_condition from kube-state-metrics; kube_node_info from kube-state-metrics but got no results
Expected
    <[]string | len:2, cap:2>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:147]: Timed out after 900.000s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:143 with:
expected ingresscontroller_info metrics from kube-state-metrics but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:147]: Timed out after 900.000s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:143 with:
expected ingresscontroller_info metrics from kube-state-metrics but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	KSM HCP Metrics metrics should be present in HCP Azure Monitoring Workspace	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-24 21:23	PR #7009	2103233919481548800
2026-09-24 18:40	PR #7009	2103192810386624512
2026-09-24 11:07	PR #7009	2103078971544440832
2026-09-24 07:37	PR #7009	2103026030053691392

Command failed after 5 attempts: oras cp arohcpsvcdev/nshneor/cs@sha256:<hex> arohcpsvcdev.azurecr.io/nshneor/cs:<hex> --from-registry-config /tmp/tmp.<id>/containers/auth.json --...

full failure pattern:

Command failed after 5 attempts: oras cp arohcpsvcdev/nshneor/cs@sha256:<hex> arohcpsvcdev.azurecr.io/nshneor/cs:<hex> --from-registry-config /tmp/tmp.<id>/containers/auth.json --to-registry-config /tmp/tmp.<id>/containers/auth.json

Signal:

provision

none

Full failure examples (2)

time=2026-09-24T12:31:58.312Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> clustersService.image.registry <nil>}:{<nil> clustersService.image.repository <nil>}@{<nil> clustersService.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-09-24T12:32:36.173Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nFetch pull secret for source registry arohcpsvcdev from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image arohcpsvcdev/nshneor/cs@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 to arohcpsvcdev.azurecr.io/nshneor/cs:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6.\nThe image will still be available under it's original digest sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 in the target registry.\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 1/5). Retrying in 2s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 2/5). Retrying in 4s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 3/5). Retrying in 8s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 4/5). Retrying in 16s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed after 5 attempts: oras cp arohcpsvcdev/nshneor/cs@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 arohcpsvcdev.azurecr.io/nshneor/cs:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 --from-registry-config /tmp/tmp.wiDcX7URTm/containers/auth.json --to-registry-config /tmp/tmp.wiDcX7URTm/containers/auth.json\n exit status 1"

time=2026-09-24T10:30:31.503Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> clustersService.image.registry <nil>}:{<nil> clustersService.image.repository <nil>}@{<nil> clustersService.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-09-24T10:31:04.673Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nFetch pull secret for source registry arohcpsvcdev from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image arohcpsvcdev/nshneor/cs@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 to arohcpsvcdev.azurecr.io/nshneor/cs:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6.\nThe image will still be available under it's original digest sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 in the target registry.\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 1/5). Retrying in 2s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 2/5). Retrying in 4s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 3/5). Retrying in 8s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed (attempt 4/5). Retrying in 16s...\nError: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: Get \"https://arohcpsvcdev/v2/nshneor/cs/manifests/sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6\": dial tcp: lookup arohcpsvcdev on 172.30.0.10:53: no such host\nCommand failed after 5 attempts: oras cp arohcpsvcdev/nshneor/cs@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 arohcpsvcdev.azurecr.io/nshneor/cs:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 --from-registry-config /tmp/tmp.bxHO86cpwB/containers/auth.json --to-registry-config /tmp/tmp.bxHO86cpwB/containers/auth.json\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/global/mirror-image	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 11:45	PR #6889	2103088386418937856
2026-09-24 09:53	PR #6889	2103060333542248448

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the ...

full failure pattern:

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'maestroMqttEventHubId'. The parameters defined in the template are: 'databaseName, eventHubId, eventhubEnabled, kustoConsumerGroupName, kustoDataConnectionName, kustoEnabled, kustoName, location'.

Signal:

provision

none

Full failure examples (2)

time=2026-09-24T23:55:56.171Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection description="Step maestroMqttDataConnection\n  Kind: ARM\n  Template: templates/maestro-mqtt-data-connection.bicep\n  Parameters: configurations/maestro-mqtt-data-connection.tmpl.bicepparam"
time=2026-09-24T23:55:56.789Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection
time=2026-09-24T23:55:57.168Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection err="failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/23db75d3c4a6f5fe32de7df35567f808137cc8acac2d401192c396cc43f1121e\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplate\",\n    \"message\": \"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'maestroMqttEventHubId'. The parameters defined in the template are: 'databaseName, eventHubId, eventhubEnabled, kustoConsumerGroupName, kustoDataConnectionName, kustoEnabled, kustoName, location'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\n    \"additionalInfo\": [\n      {\n        \"type\": \"TemplateViolation\",\n        \"info\": {\n          \"lineNumber\": 0,\n          \"linePosition\": 0,\n          \"path\": \"\"\n        }\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-24T22:47:58.457Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection description="Step maestroMqttDataConnection\n  Kind: ARM\n  Template: templates/maestro-mqtt-data-connection.bicep\n  Parameters: configurations/maestro-mqtt-data-connection.tmpl.bicepparam"
time=2026-09-24T22:47:59.030Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection
time=2026-09-24T22:47:59.240Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection err="failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/754de7cea577bd27c73251066594765f388be9580303d6f77fd3975785c55d9b\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplate\",\n    \"message\": \"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'maestroMqttEventHubId'. The parameters defined in the template are: 'databaseName, eventHubId, eventhubEnabled, kustoConsumerGroupName, kustoDataConnectionName, kustoEnabled, kustoName, location'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\n    \"additionalInfo\": [\n      {\n        \"type\": \"TemplateViolation\",\n        \"info\": {\n          \"lineNumber\": 0,\n          \"linePosition\": 0,\n          \"path\": \"\"\n        }\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/kusto/maestroMqttDataConnection	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:50	PR #7146	2103270905017798656
2026-09-24 22:27	PR #7146	2103249876442681344

VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4

full failure pattern:

VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc00141d0c8>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.33) failed: no version in clusterversion/version status.history is greater than initial "4.21.33"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc00141d0c8>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.33) failed: no version in clusterversion/version status.history is greater than initial "4.21.33"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000e85578>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.22.14) failed: no version in clusterversion/version status.history is greater than initial "4.22.14"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000e85578>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.22.14) failed: no version in clusterversion/version status.history is greater than initial "4.22.14"
    ...

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 14:18	PR #6577	2103126923872833536

ERROR CODE: DeploymentFailed; detail code HttpRequestException; detail message Error while processing network security perimeter association: An error occurred while sending the r...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code HttpRequestException; detail message Error while processing network security perimeter association: An error occurred while sending the request.; provider Microsoft.Network

Signal:

provision

none

Full failure examples (1)

time=2026-09-24T17:25:13.342Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-09-24T17:25:13.641Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-09-24T17:25:14.167Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=9d3f2ed9b958df750e3b7f101250ebe891c1a35bc943187871975b34cdf31ee3 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4687104-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9d3f2ed9b958df750e3b7f101250ebe891c1a35bc943187871975b34cdf31ee3
time=2026-09-24T17:25:44.283Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4687104-mgmt-2/providers/Microsoft.Resources/deployments/9d3f2ed9b958df750e3b7f101250ebe891c1a35bc943187871975b34cdf31ee3/operationStatuses/08584113361715956682\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4687104-mgmt-2/providers/Microsoft.Resources/deployments/profile-bmfsiknuehg4q\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4687104-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j4687104-mgmt-2/resourceAssociations/nsp-j4687104-mgmt-2-ra-d68332b2-0c36-56b4-8794-de05fd33fea0\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4687104-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j4687104-mgmt-2/resourceAssociations/nsp-j4687104-mgmt-2-ra-d68332b2-0c36-56b4-8794-de05fd33fea0, Error message: An error occurred while sending the request.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 16:41	PR #7009	2103162898384687104

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...

full failure pattern:

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceede... message truncated; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000d1d080>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-dkvsbz" in resourcegroup="rg-np-version-upgrade-dkvsbz-sql98dwv9hds" to finish creating: GET https://rp.j8866176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa5258ae-6ac0-4393-ba90-fd7fff99d1fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa5258ae-6ac0-4393-ba90-fd7fff99d1fd",
      "name": "aa5258ae-6ac0-4393-ba90-fd7fff99d1fd",
      "status": "Failed",
      "startTime": "2026-09-24T19:41:58.103375269Z",
      "endTime": "2026-09-24T20:01:00.35851472Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c8k0y6n8u7a0y5r-npupgrade-4-20-ctsc2-95r6h: Failed: interfaces failed to create or update. err: failed to get existing resource rg-np-version-upgrade-dkvsbz-sql98dwv9hds-np-upgrade-dk-d348d3ab/c8k0y6n8u7a0y5r-npupgrade-4-20-ctsc2-95r6h-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-dkvsbz-sql98dwv9hds-np-upgrade-dk-d348d3ab/providers/Microsoft.Network/networkInterfaces/c8k0y6n8u7a0y5r-npupgrade-4-20-ctsc2-95r6h-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceede... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000d1d080>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-dkvsbz" in resourcegroup="rg-np-version-upgrade-dkvsbz-sql98dwv9hds" to finish creating: GET https://rp.j8866176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa5258ae-6ac0-4393-ba90-fd7fff99d1fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aa5258ae-6ac0-4393-ba90-fd7fff99d1fd",
      "name": "aa5258ae-6ac0-4393-ba90-fd7fff99d1fd",
      "status": "Failed",
      "startTime": "2026-09-24T19:41:58.103375269Z",
      "endTime": "2026-09-24T20:01:00.35851472Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c8k0y6n8u7a0y5r-npupgrade-4-20-ctsc2-95r6h: Failed: interfaces failed to create or update. err: failed to get existing resource rg-np-version-upgrade-dkvsbz-sql98dwv9hds-np-upgrade-dk-d348d3ab/c8k0y6n8u7a0y5r-npupgrade-4-20-ctsc2-95r6h-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-dkvsbz-sql98dwv9hds-np-upgrade-dk-d348d3ab/providers/Microsoft.Network/networkInterfaces/c8k0y6n8u7a0y5r-npupgrade-4-20-ctsc2-95r6h-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceede... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 18:36	PR #7141	2103191956178866176

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...

full failure pattern:

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available. ... message truncated; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc00145fb18>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-4bnpzclpb4vn" to finish creating: GET https://rp.j9945984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae2ff5d-353f-4924-8074-8541fe3fffdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae2ff5d-353f-4924-8074-8541fe3fffdf",
      "name": "1ae2ff5d-353f-4924-8074-8541fe3fffdf",
      "status": "Failed",
      "startTime": "2026-09-24T22:34:11.896891667Z",
      "endTime": "2026-09-24T22:53:20.096362323Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-vbqjx-klzb2: Failed: virtualmachine failed to create or update. err: failed to get existing resource feature-aggregation-4bnpzclpb4vn--managed/agg-cluster-agg-np-a-vbqjx-klzb2 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-4bnpzclpb4vn--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-vbqjx-klzb2\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '07aa0e83-353e-444c-9f1d-d023ac3c5396' exceeded. Please try again after '10' seconds after additional tokens are available. ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc00145fb18>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-4bnpzclpb4vn" to finish creating: GET https://rp.j9945984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae2ff5d-353f-4924-8074-8541fe3fffdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae2ff5d-353f-4924-8074-8541fe3fffdf",
      "name": "1ae2ff5d-353f-4924-8074-8541fe3fffdf",
      "status": "Failed",
      "startTime": "2026-09-24T22:34:11.896891667Z",
      "endTime": "2026-09-24T22:53:20.096362323Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-vbqjx-klzb2: Failed: virtualmachine failed to create or update. err: failed to get existing resource feature-aggregation-4bnpzclpb4vn--managed/agg-cluster-agg-np-a-vbqjx-klzb2 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-4bnpzclpb4vn--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-vbqjx-klzb2\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '07aa0e83-353e-444c-9f1d-d023ac3c5396' exceeded. Please try again after '10' seconds after additional tokens are available. ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 21:23	PR #7135	2103233926699945984

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machine...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID ; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready Machine <node>: Creating: virtualmachine creating or updating Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000b80880>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-cpn7mwhqrdqw" to finish creating: GET https://rp.j4556544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d56b67b-43e0-4cd7-a969-5552b2f1c302
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d56b67b-43e0-4cd7-a969-5552b2f1c302",
      "name": "2d56b67b-43e0-4cd7-a969-5552b2f1c302",
      "status": "Failed",
      "startTime": "2026-09-24T20:50:39.494369799Z",
      "endTime": "2026-09-24T21:09:39.1222123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-7rv28: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-drpbc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-7rv28: Creating: virtualmachine creating or updating\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-drpbc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000b80880>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-cpn7mwhqrdqw" to finish creating: GET https://rp.j4556544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d56b67b-43e0-4cd7-a969-5552b2f1c302
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d56b67b-43e0-4cd7-a969-5552b2f1c302",
      "name": "2d56b67b-43e0-4cd7-a969-5552b2f1c302",
      "status": "Failed",
      "startTime": "2026-09-24T20:50:39.494369799Z",
      "endTime": "2026-09-24T21:09:39.1222123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-7rv28: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-drpbc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-7rv28: Creating: virtualmachine creating or updating\nMachine oidc-wi-cluster-oidc-wi-np-bm6bt-drpbc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 19:22	PR #6970	2103203513864556544

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machine...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-djkxv-<id> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-djkxv-<id> to exist; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002703dd0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-x45qskfpvk4k" to finish creating: GET https://rp.j7111680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50f10e36-adf3-41e9-a281-5a223249a720
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50f10e36-adf3-41e9-a281-5a223249a720",
      "name": "50f10e36-adf3-41e9-a281-5a223249a720",
      "status": "Failed",
      "startTime": "2026-09-24T18:57:03.302695229Z",
      "endTime": "2026-09-24T19:16:10.982967273Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-djkxv-44p2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-x45qskfpvk4k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-djkxv-44p2x to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-djkxv-44p2x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-x45qskfpvk4k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-djkxv-44p2x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002703dd0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-x45qskfpvk4k" to finish creating: GET https://rp.j7111680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50f10e36-adf3-41e9-a281-5a223249a720
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50f10e36-adf3-41e9-a281-5a223249a720",
      "name": "50f10e36-adf3-41e9-a281-5a223249a720",
      "status": "Failed",
      "startTime": "2026-09-24T18:57:03.302695229Z",
      "endTime": "2026-09-24T19:16:10.982967273Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-djkxv-44p2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-x45qskfpvk4k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-djkxv-44p2x to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-djkxv-44p2x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-x45qskfpvk4k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-djkxv-44p2x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 18:00	PR #6993	2103182792887111680

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 2, want >= 3 (autoscaling min); node pool condition AllNodesHealthy is ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 2, want >= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0017d2100>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-kmp4pw976cvp" to finish creating: GET https://rp.j4556544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fe3f92f-008e-475a-a316-514fa9d56551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fe3f92f-008e-475a-a316-514fa9d56551",
      "name": "4fe3f92f-008e-475a-a316-514fa9d56551",
      "status": "Failed",
      "startTime": "2026-09-24T20:45:04.211205802Z",
      "endTime": "2026-09-24T21:04:09.101063234Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kmp4pw976cvp--managed/providers/Microsoft.Compute/virtualMachines/l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kmp4pw976cvp--managed/providers/Microsoft.Compute/virtualMachines/l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0017d2100>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-kmp4pw976cvp" to finish creating: GET https://rp.j4556544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fe3f92f-008e-475a-a316-514fa9d56551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fe3f92f-008e-475a-a316-514fa9d56551",
      "name": "4fe3f92f-008e-475a-a316-514fa9d56551",
      "status": "Failed",
      "startTime": "2026-09-24T20:45:04.211205802Z",
      "endTime": "2026-09-24T21:04:09.101063234Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kmp4pw976cvp--managed/providers/Microsoft.Compute/virtualMachines/l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-kmp4pw976cvp--managed/providers/Microsoft.Compute/virtualMachines/l7c9r0y4z3t5x4s-autoscale-noaz-9blrl-6wcfm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 19:22	PR #6970	2103203513864556544

failed to create customer-infra: failed waiting for deployment "customer-infra-autoscaling-hcp-cluster-<id>" in resourcegroup="<resource-group>" to finish: context deadline exceed...

full failure pattern:

failed to create customer-infra: failed waiting for deployment "customer-infra-autoscaling-hcp-cluster-<id>" in resourcegroup="<resource-group>" to finish: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create customer resources for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ac8540>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-autoscaling-hcp-cluster-pjz7p4" in resourcegroup="autoscaling-cluster-7qnnh4q5h8fk" to finish: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create customer resources for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ac8540>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-autoscaling-hcp-cluster-pjz7p4" in resourcegroup="autoscaling-cluster-7qnnh4q5h8fk" to finish: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 14:55	PR #7135	2103136210296770560

failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1

Signal:

provision

none

Full failure examples (1)

time=2026-09-24T23:38:34.810Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 description="Step mitigations\n Kind: Helm\n"
time=2026-09-24T23:38:35.282Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4]"
time=2026-09-24T23:38:35.282Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-09-24T23:38:35.513Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 namespace=aro-hcp-mitigations
time=2026-09-24T23:38:35.703Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 dryRun=false
time=2026-09-24T23:38:35.703Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-09-24T23:38:35.755Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-09-24T23:39:19.239Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-09-24T23:39:19.239Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/mitigations (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:03	PR #7133	2103259159142076416

failed to roll out Helm release: resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17 context deadline exceeded

full failure pattern:

failed to roll out Helm release: resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17 context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-09-24T16:33:44.590Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n"
time=2026-09-24T16:33:45.122Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 values="map[captureMode:all clusterName:ci01-j1695232-mgmt-2 enabled:true environment:ci01 image:map[digest:sha256:21861cc03f51cd5c5fcc5f37f79aa945bfe633b1fbdb5c662d6376edb84a8a45 registry:arohcpsvcdev.azurecr.io repository:ci-op-s7p7jyyz/pipeline] region:canadacentral resources:map[limits:map[memory:256Mi] requests:map[cpu:20m memory:64Mi]] serviceAccount:map[name:swift-recorder]]"
time=2026-09-24T16:33:45.122Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-09-24T16:33:45.293Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 namespace=swift-recorder
time=2026-09-24T16:33:45.465Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 dryRun=false
time=2026-09-24T16:33:45.465Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-09-24T16:33:45.516Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-09-24T16:43:47.867Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17\ncontext deadline exceeded"
time=2026-09-24T16:43:47.867Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.SwiftRecorder/management/deploy (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 15:35	PR #7117	2103146302601695232

pod targeting cordoned node <node> should not be assigned a node

full failure pattern:

pod targeting cordoned node <node> should not be assigned a node

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:273]: Failed after 0.060s.
pod on cordoned node x6j5o0j5b0f4m6v-cordon-np-xzf6f-b82cp should remain Pending
The function passed to Consistently failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:269 with:
pod targeting cordoned node x6j5o0j5b0f4m6v-cordon-np-xzf6f-b82cp should not be assigned a node
Expected
    <string>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:273]: Failed after 0.060s.
pod on cordoned node x6j5o0j5b0f4m6v-cordon-np-xzf6f-b82cp should remain Pending
The function passed to Consistently failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:269 with:
pod targeting cordoned node x6j5o0j5b0f4m6v-cordon-np-xzf6f-b82cp should not be assigned a node
Expected
    <string>: ...
to be empty

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:21	PR #6922	2103006860184391680

public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host

full failure pattern:

public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:216]: Timed out after 600.001s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:214 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud": dial tcp: lookup console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host
Unexpected error:
    <*url.Error | 0xc001758f60>: 
    Get "https://console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud": dial tcp: lookup console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:216]: Timed out after 600.001s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:214 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud": dial tcp: lookup console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host
Unexpected error:
    <*url.Error | 0xc001758f60>: 
    Get "https://console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud": dial tcp: lookup console-openshift-console.apps.aro.private-kas.fpqv.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 00:11	PR #7112	2102913782756937728

--------------------------------------------------------------------------------

full failure pattern:

--------------------------------------------------------------------------------

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gqsdx8vgs4cm" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020af38d-4caa-4310-a595-881bec9e9405
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020af38d-4caa-4310-a595-881bec9e9405",
  "name": "020af38d-4caa-4310-a595-881bec9e9405",
  "status": "Failed",
  "startTime": "2026-09-24T10:30:43.520242825Z",
  "endTime": "2026-09-24T10:31:20.235327414Z",
  "error": {
    "code": "InternalServerError",
    "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
  }
}
--------------------------------------------------------------------------------
 failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gqsdx8vgs4cm" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edfe1588-77de-4f6d-a724-94f5c5b56313
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edfe1588-77de-4f6d-a724-94f5c5b56313",
  "name": "edfe1588-77de-4f6d-a724-94f5c5b56313",
  "status": "Failed",
  "startTime": "2026-09-24T10:30:43.575931644Z",
  "endTime": "2026-09-24T10:31:20.247783059Z",
  "error": {
    "code": "InternalServerError",
    "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gqsdx8vgs4cm" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020af38d-4caa-4310-a595-881bec9e9405
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020af38d-4caa-4310-a595-881bec9e9405",
  "name": "020af38d-4caa-4310-a595-881bec9e9405",
  "status": "Failed",
  "startTime": "2026-09-24T10:30:43.520242825Z",
  "endTime": "2026-09-24T10:31:20.235327414Z",
  "error": {
    "code": "InternalServerError",
    "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
  }
}
--------------------------------------------------------------------------------
 failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gqsdx8vgs4cm" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edfe1588-77de-4f6d-a724-94f5c5b56313
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edfe1588-77de-4f6d-a724-94f5c5b56313",
  "name": "edfe1588-77de-4f6d-a724-94f5c5b56313",
  "status": "Failed",
  "startTime": "2026-09-24T10:30:43.575931644Z",
  "endTime": "2026-09-24T10:31:20.247783059Z",
  "error": {
    "code": "InternalServerError",
    "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:2, cap:2>: ...
to be empty

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 09:33	PR #6917	2103055129497309184

alert [svc] ServiceMemoryDrift fired

full failure pattern:

alert [svc] ServiceMemoryDrift fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T18:08:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4417536-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2fbf2", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2fbf2 (namespace arobit) on cluster ci01-j4417536-svc has averaged 125.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:41:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1914624-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8vzwj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8vzwj (namespace arobit) on cluster ci01-j1914624-svc has averaged 126.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:49:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1279232-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5zn5h", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-5zn5h (namespace arobit) on cluster ci01-j1279232-svc has averaged 125.1% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire	28

Affected runs (28)

Date (UTC)	Associated PR	Prow job
2026-09-24 16:09	PR #6970	2103154812324417536
2026-09-24 15:35	PR #7045	2103146310981914624
2026-09-24 15:32	PR #6917	2103145551821279232
2026-09-24 15:08	PR #7053	2103139552272060416
2026-09-24 14:55	PR #7135	2103136210296770560
2026-09-24 13:28	PR #6915	2103114369016532992
2026-09-24 13:12	PR #7131	2103110260871925760
2026-09-24 12:56	PR #7132	2103106328183443456
2026-09-24 12:28	PR #6917	2103099181529829376
2026-09-24 12:28	PR #6922	2103099182641319936
2026-09-24 11:47	PR #7123	2103089013568049152
2026-09-24 11:07	PR #7009	2103078971544440832
2026-09-24 09:33	PR #6922	2103055146278719488
2026-09-24 09:33	PR #6917	2103055129497309184
2026-09-24 09:33	PR #6226	2103055124225069056
2026-09-24 09:33	PR #7127	2103055116536909824
2026-09-24 08:12	PR #7045	2103034733737807872
2026-09-24 07:37	PR #7009	2103026030053691392
2026-09-24 07:33	PR #6995	2103024914473684992
2026-09-24 06:37	PR #7126	2103010871180529664
2026-09-24 06:00	PR #7105	2103001507350712320
2026-09-24 05:11	PR #5562	2102989394158817280
2026-09-24 03:22	PR #7112	2102961799233015808
2026-09-24 00:41	PR #7115	2102921426674651136
2026-09-24 00:20	PR #7054	2102916121228218368
2026-09-24 00:12	PR #7123	2102914094729269248
2026-09-24 00:11	PR #7112	2102913782756937728
2026-09-24 00:08	PR #6846	2102912928209440768

alert [svc] FrontendPathMedianLatency fired

full failure pattern:

alert [svc] FrontendPathMedianLatency fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T00:29:48Z
  Ended:       2026-09-25T01:02:49Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6834176-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6834176-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T00:06:48Z
  Ended:       2026-09-25T00:12:51Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4729344-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4729344-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T22:56:37Z
  Ended:       2026-09-24T23:04:39Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9945984-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9945984-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire	24

Affected runs (24)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:44	PR #6969	2103269477926834176
2026-09-24 22:49	PR #7145	2103255589084729344
2026-09-24 21:23	PR #7135	2103233926699945984
2026-09-24 21:23	PR #7009	2103233919481548800
2026-09-24 21:21	PR #7068	2103233417079427072
2026-09-24 20:23	PR #7053	2103218829420138496
2026-09-24 18:40	PR #7135	2103192892754366464
2026-09-24 18:40	PR #7068	2103192838828199936
2026-09-24 17:12	PR #7117	2103170613366493184
2026-09-24 16:36	PR #7123	2103161541376348160
2026-09-24 16:09	PR #7068	2103155003655983104
2026-09-24 16:09	PR #6970	2103154812324417536
2026-09-24 15:35	PR #7045	2103146310981914624
2026-09-24 14:55	PR #7135	2103136210296770560
2026-09-24 13:12	PR #7131	2103110260871925760
2026-09-24 13:09	PR #6746	2103109521260941312
2026-09-24 09:33	PR #6917	2103055129497309184
2026-09-24 09:33	PR #7112	2103055091593383936
2026-09-24 06:37	PR #7126	2103010871180529664
2026-09-24 05:11	PR #5562	2102989394158817280
2026-09-24 01:20	PR #6226	2102931134991044608
2026-09-24 00:41	PR #7115	2102921426674651136
2026-09-24 00:16	PR #7119	2102915118240108544
2026-09-24 00:11	PR #7112	2102913782756937728

alert [svc] ServiceCPUDrift fired

full failure pattern:

alert [svc] ServiceCPUDrift fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T01:15:33Z
  Ended:       2026-09-25T01:24:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0596736-mgmt-2", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-646d8d7d48-fmq9z", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-646d8d7d48-fmq9z (namespace kube-applier) on cluster ci01-j0596736-mgmt-2 has averaged 129.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T00:08:21Z
  Ended:       2026-09-25T00:18:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9877248-mgmt-1", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-7487b8fc78-kvh8n", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-7487b8fc78-kvh8n (namespace kube-applier) on cluster ci01-j9877248-mgmt-1 has averaged 133.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T23:55:38Z
  Ended:       2026-09-25T00:01:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156224-mgmt-1", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-75497856bf-g2n4k", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-75497856bf-g2n4k (namespace kube-applier) on cluster ci01-j9156224-mgmt-1 has averaged 122.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ServiceCPUDrift does not fire	15

Affected runs (15)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:49	PR #6846	2103270655540596736
2026-09-24 22:38	PR #7123	2103252727629877248
2026-09-24 22:18	PR #7148	2103247783359156224
2026-09-24 22:08	PR #7059	2103245132269621248
2026-09-24 21:23	PR #7135	2103233926699945984
2026-09-24 21:23	PR #7009	2103233919481548800
2026-09-24 21:21	PR #7068	2103233417079427072
2026-09-24 20:23	PR #6993	2103218872113958912
2026-09-24 19:22	PR #6970	2103203513864556544
2026-09-24 19:21	PR #6866	2103203214361890816
2026-09-24 19:09	PR #7123	2103200191157899264
2026-09-24 19:09	PR #6821	2103200143447691264
2026-09-24 18:40	PR #7135	2103192892754366464
2026-09-24 18:40	PR #7068	2103192838828199936
2026-09-24 17:32	PR #7053	2103175794774773760

ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid

full failure pattern:

ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid

Signal:

provision

none

Full failure examples (3)

time=2026-09-24T23:48:24.249Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:48:25.676Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:48:26.816Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9fe92f2075d3a162115703f2a2dd0a40f461263d369dfb42681221c1b82cb479 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1648000%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9fe92f2075d3a162115703f2a2dd0a40f461263d369dfb42681221c1b82cb479
time=2026-09-24T23:50:57.493Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1648000/providers/Microsoft.Resources/deployments/9fe92f2075d3a162115703f2a2dd0a40f461263d369dfb42681221c1b82cb479/operationStatuses/08584113131791545522\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1648000/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1648000/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1648000\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report a6a8d0ae-c018-4639-a518-3ed1c7beb811:9/24/2026 11:49:54 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-24T23:52:57.493Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:52:58.386Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:52:59.129Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=cf6e37d6cff770de12dbdbffa8c4623b3a737efd76d033bcf339dec1a4fb04e1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1648000%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcf6e37d6cff770de12dbdbffa8c4623b3a737efd76d033bcf339dec1a4fb04e1
time=2026-09-25T00:09:04.051Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1648000/providers/Microsoft.Resources/deployments/cf6e37d6cff770de12dbdbffa8c4623b3a737efd76d033bcf339dec1a4fb04e1/operationStatuses/08584113129068388577\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1648000/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1648000/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1648000\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report fc9f1334-cb66-4a1b-a69a-b44d3f0d15c9:9/25/2026 12:08:37 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-25T00:11:04.059Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-25T00:11:04.960Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-25T00:11:05.763Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=a4eb99266d655f9a85d7de315dc769d2d5008a2127b29773f350744d3fa412d7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1648000%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fa4eb99266d655f9a85d7de315dc769d2d5008a2127b29773f350744d3fa412d7
time=2026-09-25T00:18:24.009Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

time=2026-09-24T23:03:28.056Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:03:29.473Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:03:30.301Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=12f2d669fa06dd4de2ba8409d1e9b7aa7f63bae2d0c78ba209c8d1082b4c957e portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1689984%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F12f2d669fa06dd4de2ba8409d1e9b7aa7f63bae2d0c78ba209c8d1082b4c957e
time=2026-09-24T23:06:00.848Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1689984/providers/Microsoft.Resources/deployments/12f2d669fa06dd4de2ba8409d1e9b7aa7f63bae2d0c78ba209c8d1082b4c957e/operationStatuses/08584113158756314414\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1689984/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1689984/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1689984\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 6bb9bafd-0fb4-4aa8-a6e2-b2e4391847ba:9/24/2026 11:04:31 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-24T23:08:00.850Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:08:01.497Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:08:02.303Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=4b6dbf996876bb5363df66ec3bbff19c39d2fbac30f092592bf67f3fff9b6f47 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1689984%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4b6dbf996876bb5363df66ec3bbff19c39d2fbac30f092592bf67f3fff9b6f47
time=2026-09-24T23:24:36.647Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1689984/providers/Microsoft.Resources/deployments/4b6dbf996876bb5363df66ec3bbff19c39d2fbac30f092592bf67f3fff9b6f47/operationStatuses/08584113156036989890\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1689984/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1689984/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1689984\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 7e35e8ae-d5b1-4064-b001-fb471f9768fb:9/24/2026 11:23:59 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-24T23:26:36.652Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:26:37.173Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:26:37.799Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=1b5feed18dc2aa2598959e31b9190f5f385e888a1a154d00d07d2ff191153df4 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1689984%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F1b5feed18dc2aa2598959e31b9190f5f385e888a1a154d00d07d2ff191153df4
time=2026-09-24T23:33:27.754Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

time=2026-09-24T23:03:18.918Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:03:20.600Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:03:21.313Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d81bee64ce02324faba35ebcfefbaa384d773cf4fc43c777c70f8e291440954b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1382656%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd81bee64ce02324faba35ebcfefbaa384d773cf4fc43c777c70f8e291440954b
time=2026-09-24T23:05:52.100Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1382656/providers/Microsoft.Resources/deployments/d81bee64ce02324faba35ebcfefbaa384d773cf4fc43c777c70f8e291440954b/operationStatuses/08584113158845641130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1382656/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1382656/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1382656\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 1a3e1a3d-0894-450c-b8f5-f4aae39286df:9/24/2026 11:04:34 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-24T23:07:52.108Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:07:53.036Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:07:53.967Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=e77ffa30236038ba6be6aa6728547670b7fd78c60b95d6556ff96ad801ee7bf4 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1382656%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fe77ffa30236038ba6be6aa6728547670b7fd78c60b95d6556ff96ad801ee7bf4
time=2026-09-24T23:24:01.808Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1382656/providers/Microsoft.Resources/deployments/e77ffa30236038ba6be6aa6728547670b7fd78c60b95d6556ff96ad801ee7bf4/operationStatuses/08584113156121186022\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1382656/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1382656/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1382656\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report f12586db-dfe7-4f9a-a86d-256d3a24ca8b:9/24/2026 11:23:57 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-24T23:26:01.815Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-24T23:26:02.592Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-24T23:26:03.555Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=8cd82016b5e13f611cfcadf8f8916eda701f48b5be5c35226e79dda3dd59a43e portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1382656%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8cd82016b5e13f611cfcadf8f8916eda701f48b5be5c35226e79dda3dd59a43e
time=2026-09-24T23:33:18.520Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra	11

Affected runs (11)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:44	PR #7118	2103269397211648000
2026-09-24 22:51	PR #7150	2103255953611689984
2026-09-24 22:38	PR #6821	2103252721661382656
2026-09-24 22:12	PR #7149	2103246251251208192
2026-09-24 21:18	PR #6359	2103232602474287104
2026-09-24 20:02	PR #6969	2103213571209957376
2026-09-24 19:31	PR #7142	2103205737139277824
2026-09-24 19:02	PR #6915	2103198517097926656
2026-09-24 18:13	PR #6920	2103186015765139456
2026-09-24 17:56	PR #6915	2103181727441620992
2026-09-24 17:44	PR #7054	2103178709123993600

alert [svc] FrontendPathLatency fired

full failure pattern:

alert [svc] FrontendPathLatency fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T17:31:32Z
  Ended:       2026-09-24T17:39:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j1914624-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j1914624-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T12:40:11Z
  Ended:       2026-09-24T12:46:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4440832-svc", component="frontend", method="post", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential", severity="warning"
  Description: More than 5% of frontend requests for post /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential exceeded 1 second over the past 30 minutes (cluster ci01-j4440832-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T08:24:07Z
  Ended:       2026-09-24T08:31:08Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j2442112-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j2442112-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-09-24 15:35	PR #7045	2103146310981914624
2026-09-24 11:07	PR #7009	2103078971544440832
2026-09-24 06:55	PR #6884	2103015428132442112
2026-09-24 05:11	PR #5562	2102989394158817280
2026-09-24 00:41	PR #7115	2102921426674651136

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T20:33:02Z
  Ended:       2026-09-24T21:02:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4556544-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t1jgr6a3uevc16juiqil4ndk5set1ce-o5o5w3j3g5c4j6m", pod="router-66cb8fd48d-wgc7r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t1jgr6a3uevc16juiqil4ndk5set1ce-o5o5w3j3g5c4j6m/router-66cb8fd48d-wgc7r has been in a non-ready state for longer than 5 minutes.

alert fired 12 time(s) (1 unknown, 11 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T19:41:45Z
  Ended:       2026-09-24T20:10:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8866176-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t1io3vu7p3gua60c6258horequmjsus-f3a9j8b5h7f1o0j", pod="router-5cc57c6448-t4p5n", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t1io3vu7p3gua60c6258horequmjsus-f3a9j8b5h7f1o0j/router-5cc57c6448-t4p5n has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T15:02:27Z
  Ended:       2026-09-24T15:08:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3943808-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t1ecumio2t0so7i2erb9l3i7dm6lu3m-sre-hcp-cluster", pod="kube-apiserver-5977bd7447-4zvrr", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t1ecumio2t0so7i2erb9l3i7dm6lu3m-sre-hcp-cluster/kube-apiserver-5977bd7447-4zvrr has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-24 19:22	PR #6970	2103203513864556544
2026-09-24 18:36	PR #7141	2103191956178866176
2026-09-24 13:15	PR #7130	2103111106493943808
2026-09-24 00:35	PR #6917	2102919814468079616

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

full failure pattern:

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001ea9b70>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001ea9b70>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001cb6930>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001cb6930>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000bbd950>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000bbd950>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-24 20:45	PR #6364	2103224244262408192
2026-09-24 11:05	PR #7045	2103078389232439296
2026-09-24 06:54	PR #7127	2103015078914691072
2026-09-24 06:37	PR #7126	2103010871180529664

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:24:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-fw4cp", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-fw4cp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:24:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-dfd7c", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-dfd7c has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:24:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-vflzc", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-vflzc has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:24:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-fn4cr", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-fn4cr has been in a non-ready state for longer than 5 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T16:38:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1695232-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-40199610-vmss000000-j9kbh", region="canadacentral", severity="warning"
  Description: Pod kube-system/eraser-aks-system-40199610-vmss000000-j9kbh has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-24T16:38:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1695232-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-nffxs", region="canadacentral", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-nffxs has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-24T16:38:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1695232-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-2szd2", region="canadacentral", severity="warning"
  Description: Pod kube-system/ama-metrics-node-2szd2 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-24T16:40:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1695232-mgmt-2", component="kubernetes-infrastructure", namespace="swift-recorder", pod="swift-recorder-tw7kk", region="canadacentral", severity="warning"
  Description: Pod swift-recorder/swift-recorder-tw7kk has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T07:42:23Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-749558849-6dgkt", region="westus3", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-749558849-6dgkt has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-24 16:34	PR #6820	2103161107324604416
2026-09-24 15:35	PR #7117	2103146302601695232
2026-09-24 06:52	PR #6503	2103014767818969088
2026-09-24 00:43	PR #7090	2102921738877669376

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...

full failure pattern:

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000a31f40>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-dlxzmd" in resourcegroup="rg-np-skip-minor-dlxzmd-8tb8jspnqx48" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2e06418b-9c11-492d-8bf1-bc7db3b3afdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2e06418b-9c11-492d-8bf1-bc7db3b3afdf",
      "name": "2e06418b-9c11-492d-8bf1-bc7db3b3afdf",
      "status": "Failed",
      "startTime": "2026-09-24T10:30:31.115048125Z",
      "endTime": "2026-09-24T10:31:10.228268414Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000a31f40>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-dlxzmd" in resourcegroup="rg-np-skip-minor-dlxzmd-8tb8jspnqx48" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2e06418b-9c11-492d-8bf1-bc7db3b3afdf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2e06418b-9c11-492d-8bf1-bc7db3b3afdf",
      "name": "2e06418b-9c11-492d-8bf1-bc7db3b3afdf",
      "status": "Failed",
      "startTime": "2026-09-24T10:30:31.115048125Z",
      "endTime": "2026-09-24T10:31:10.228268414Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-d4v522" for cluster "cluster-candidate-4-22-d4v522"
Unexpected error:
    <*fmt.wrapError | 0xc0008ad540>: 
    failed to create NodePool np-d4v522: failed waiting for nodepool="np-d4v522" for cluster "cluster-candidate-4-22-d4v522" in resourcegroup="rg-candidate-4-22-q2jqvkjfll66" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a57495d-192b-47f3-8f08-3dbd939c2ccc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a57495d-192b-47f3-8f08-3dbd939c2ccc",
      "name": "9a57495d-192b-47f3-8f08-3dbd939c2ccc",
      "status": "Failed",
      "startTime": "2026-09-24T10:30:51.31445342Z",
      "endTime": "2026-09-24T10:31:30.134898766Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-d4v522" for cluster "cluster-candidate-4-22-d4v522"
Unexpected error:
    <*fmt.wrapError | 0xc0008ad540>: 
    failed to create NodePool np-d4v522: failed waiting for nodepool="np-d4v522" for cluster "cluster-candidate-4-22-d4v522" in resourcegroup="rg-candidate-4-22-q2jqvkjfll66" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a57495d-192b-47f3-8f08-3dbd939c2ccc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a57495d-192b-47f3-8f08-3dbd939c2ccc",
      "name": "9a57495d-192b-47f3-8f08-3dbd939c2ccc",
      "status": "Failed",
      "startTime": "2026-09-24T10:30:51.31445342Z",
      "endTime": "2026-09-24T10:31:30.134898766Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000f01380>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-bpr946thzk52" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/62555631-7ec6-4b02-b164-698c4d6f90b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/62555631-7ec6-4b02-b164-698c4d6f90b4",
      "name": "62555631-7ec6-4b02-b164-698c4d6f90b4",
      "status": "Failed",
      "startTime": "2026-09-24T10:24:31.78038551Z",
      "endTime": "2026-09-24T10:25:09.920364245Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000f01380>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-bpr946thzk52" to finish creating: GET https://rp.j7309184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/62555631-7ec6-4b02-b164-698c4d6f90b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/62555631-7ec6-4b02-b164-698c4d6f90b4",
      "name": "62555631-7ec6-4b02-b164-698c4d6f90b4",
      "status": "Failed",
      "startTime": "2026-09-24T10:24:31.78038551Z",
      "endTime": "2026-09-24T10:25:09.920364245Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (7)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 09:33	PR #6917	2103055129497309184
2026-09-24 00:16	PR #7119	2102915118240108544

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-c44pd8c2jtgp/cluster-candidate-4-21-k4rdfg should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010bc2e0>: 
    failed to create HCP cluster cluster-candidate-4-21-k4rdfg: failed waiting for cluster="cluster-candidate-4-21-k4rdfg" in resourcegroup="rg-candidate-4-21-c44pd8c2jtgp" to finish creating: GET https://rp.j4556544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c4682523-7a01-4560-b35e-4fc4d15613ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c4682523-7a01-4560-b35e-4fc4d15613ae",
      "name": "c4682523-7a01-4560-b35e-4fc4d15613ae",
      "status": "Failed",
      "startTime": "2026-09-24T20:22:03.886308121Z",
      "endTime": "2026-09-24T20:51:09.533160535Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-c44pd8c2jtgp/cluster-candidate-4-21-k4rdfg should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010bc2e0>: 
    failed to create HCP cluster cluster-candidate-4-21-k4rdfg: failed waiting for cluster="cluster-candidate-4-21-k4rdfg" in resourcegroup="rg-candidate-4-21-c44pd8c2jtgp" to finish creating: GET https://rp.j4556544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c4682523-7a01-4560-b35e-4fc4d15613ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c4682523-7a01-4560-b35e-4fc4d15613ae",
      "name": "c4682523-7a01-4560-b35e-4fc4d15613ae",
      "status": "Failed",
      "startTime": "2026-09-24T20:22:03.886308121Z",
      "endTime": "2026-09-24T20:51:09.533160535Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-c7xkx7wln57r/cluster-candidate-5-0-87wr9p should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010cc280>: 
    failed to create HCP cluster cluster-candidate-5-0-87wr9p: failed waiting for cluster="cluster-candidate-5-0-87wr9p" in resourcegroup="rg-candidate-5-0-c7xkx7wln57r" to finish creating: GET https://rp.j8866176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9d1e9a3-4f45-45e2-a7a8-d0765c81f904
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9d1e9a3-4f45-45e2-a7a8-d0765c81f904",
      "name": "d9d1e9a3-4f45-45e2-a7a8-d0765c81f904",
      "status": "Failed",
      "startTime": "2026-09-24T19:29:18.072972108Z",
      "endTime": "2026-09-24T19:58:19.977586305Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 22m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-c7xkx7wln57r/cluster-candidate-5-0-87wr9p should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010cc280>: 
    failed to create HCP cluster cluster-candidate-5-0-87wr9p: failed waiting for cluster="cluster-candidate-5-0-87wr9p" in resourcegroup="rg-candidate-5-0-c7xkx7wln57r" to finish creating: GET https://rp.j8866176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9d1e9a3-4f45-45e2-a7a8-d0765c81f904
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9d1e9a3-4f45-45e2-a7a8-d0765c81f904",
      "name": "d9d1e9a3-4f45-45e2-a7a8-d0765c81f904",
      "status": "Failed",
      "startTime": "2026-09-24T19:29:18.072972108Z",
      "endTime": "2026-09-24T19:58:19.977586305Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 22m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 19:22	PR #6970	2103203513864556544
2026-09-24 18:36	PR #7141	2103191956178866176

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD, STG

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000721c20>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8nk2s8" in resourcegroup="rg-np-noedge-8nk2s8-pbf4w4lpxflj" to finish creating: GET https://rp.j9067520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0",
      "name": "bf579354-82a6-4511-b470-a74098b9d7f0",
      "status": "Failed",
      "startTime": "2026-09-24T13:38:51.627589072Z",
      "endTime": "2026-09-24T13:57:59.764961372Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000721c20>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8nk2s8" in resourcegroup="rg-np-noedge-8nk2s8-pbf4w4lpxflj" to finish creating: GET https://rp.j9067520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0",
      "name": "bf579354-82a6-4511-b470-a74098b9d7f0",
      "status": "Failed",
      "startTime": "2026-09-24T13:38:51.627589072Z",
      "endTime": "2026-09-24T13:57:59.764961372Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00100c160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-wjm85h57dqlz" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921",
      "name": "deca3434-569c-4feb-abcd-1a0dae06f921",
      "status": "Failed",
      "startTime": "2026-09-24T08:38:16.231035723Z",
      "endTime": "2026-09-24T08:57:18.997473969Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00100c160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-wjm85h57dqlz" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921",
      "name": "deca3434-569c-4feb-abcd-1a0dae06f921",
      "status": "Failed",
      "startTime": "2026-09-24T08:38:16.231035723Z",
      "endTime": "2026-09-24T08:57:18.997473969Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 12:25	PR #7065	2103098516569067520
2026-09-24 07:33	PR #6995	2103024914473684992

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T16:50:03Z
  Ended:       2026-09-24T16:57:05Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1914624-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T04:25:52Z
  Ended:       2026-09-24T04:43:55Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3233536-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 15:35	PR #7045	2103146310981914624
2026-09-24 03:38	PR #6917	2102965797063233536

Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again

full failure pattern:

Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again

Signal:

e2e

PROD

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node k8w2j3t8n4o7w6y-cordon-np-fqrz9-2zs4k
Unexpected error:
    <*errors.StatusError | 0xc0012d1400>: 
    Operation cannot be fulfilled on nodes "k8w2j3t8n4o7w6y-cordon-np-fqrz9-2zs4k": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node k8w2j3t8n4o7w6y-cordon-np-fqrz9-2zs4k
Unexpected error:
    <*errors.StatusError | 0xc0012d1400>: 
    Operation cannot be fulfilled on nodes "k8w2j3t8n4o7w6y-cordon-np-fqrz9-2zs4k": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:214]: failed to cordon node a2u7a3d0r9z7j0c-cordon-np-tvtlh-5pbth
Unexpected error:
    <*errors.StatusError | 0xc000a01cc0>: 
    Operation cannot be fulfilled on nodes "a2u7a3d0r9z7j0c-cordon-np-tvtlh-5pbth": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:214]: failed to cordon node a2u7a3d0r9z7j0c-cordon-np-tvtlh-5pbth
Unexpected error:
    <*errors.StatusError | 0xc000a01cc0>: 
    Operation cannot be fulfilled on nodes "a2u7a3d0r9z7j0c-cordon-np-tvtlh-5pbth": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 12:28	PR #6226	2103099155617419264
2026-09-24 07:33	PR #6995	2103024914473684992

alert [svc] MaestroAgentMetricsProxyDown fired

full failure pattern:

alert [svc] MaestroAgentMetricsProxyDown fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T19:45:14Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j6380032-mgmt-1", component="monitoring-infrastructure", region="canadacentral", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T16:29:24Z
  Ended:       2026-09-24T16:38:26Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j1695232-mgmt-1", component="monitoring-infrastructure", region="canadacentral", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 18:40	PR #7045	2103192859166380032
2026-09-24 15:35	PR #7117	2103146302601695232

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

PROD

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003790c60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2tslxd58w56h" to finish creating: GET https://rp.j9945984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bbdcadb-3a21-4369-a3b9-263a4eb93a49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bbdcadb-3a21-4369-a3b9-263a4eb93a49",
      "name": "3bbdcadb-3a21-4369-a3b9-263a4eb93a49",
      "status": "Failed",
      "startTime": "2026-09-24T22:33:47.939426596Z",
      "endTime": "2026-09-24T22:52:50.218624914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-2tslxd58w56h--managed/providers/Microsoft.Compute/virtualMachines/j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-2tslxd58w56h--managed/providers/Microsoft.Compute/virtualMachines/j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003790c60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2tslxd58w56h" to finish creating: GET https://rp.j9945984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bbdcadb-3a21-4369-a3b9-263a4eb93a49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bbdcadb-3a21-4369-a3b9-263a4eb93a49",
      "name": "3bbdcadb-3a21-4369-a3b9-263a4eb93a49",
      "status": "Failed",
      "startTime": "2026-09-24T22:33:47.939426596Z",
      "endTime": "2026-09-24T22:52:50.218624914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-2tslxd58w56h--managed/providers/Microsoft.Compute/virtualMachines/j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-2tslxd58w56h--managed/providers/Microsoft.Compute/virtualMachines/j9h7b2t4y3p0g4b-np-delete-main-vngst-vfmpn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003143460>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dtj57jh59wlh" to finish creating: GET https://rp.j6624512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7eb325fa-7f6c-4e10-8c9b-3bebf9248544
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7eb325fa-7f6c-4e10-8c9b-3bebf9248544",
      "name": "7eb325fa-7f6c-4e10-8c9b-3bebf9248544",
      "status": "Failed",
      "startTime": "2026-09-24T19:46:17.078177969Z",
      "endTime": "2026-09-24T20:05:23.632105826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dtj57jh59wlh--managed/providers/Microsoft.Compute/virtualMachines/l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dtj57jh59wlh--managed/providers/Microsoft.Compute/virtualMachines/l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003143460>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dtj57jh59wlh" to finish creating: GET https://rp.j6624512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7eb325fa-7f6c-4e10-8c9b-3bebf9248544
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7eb325fa-7f6c-4e10-8c9b-3bebf9248544",
      "name": "7eb325fa-7f6c-4e10-8c9b-3bebf9248544",
      "status": "Failed",
      "startTime": "2026-09-24T19:46:17.078177969Z",
      "endTime": "2026-09-24T20:05:23.632105826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dtj57jh59wlh--managed/providers/Microsoft.Compute/virtualMachines/l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dtj57jh59wlh--managed/providers/Microsoft.Compute/virtualMachines/l6n2u5f4n0l6i5m-np-delete-main-lqq79-j492z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-24 21:23	PR #7135	2103233926699945984
2026-09-24 18:40	PR #7009	2103192810386624512

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0xc000687d40>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-jl6ktm48b4wh" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0fce08-88e2-4170-ac88-340cc76559e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0fce08-88e2-4170-ac88-340cc76559e9",
      "name": "1a0fce08-88e2-4170-ac88-340cc76559e9",
      "status": "Failed",
      "startTime": "2026-09-24T07:41:08.812973457Z",
      "endTime": "2026-09-24T08:10:12.41059037Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0xc000687d40>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-jl6ktm48b4wh" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0fce08-88e2-4170-ac88-340cc76559e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0fce08-88e2-4170-ac88-340cc76559e9",
      "name": "1a0fce08-88e2-4170-ac88-340cc76559e9",
      "status": "Failed",
      "startTime": "2026-09-24T07:41:08.812973457Z",
      "endTime": "2026-09-24T08:10:12.41059037Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000fced80>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-tmsknb268d6w" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/03151659-c9c2-4800-86d8-c833276deaec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/03151659-c9c2-4800-86d8-c833276deaec",
      "name": "03151659-c9c2-4800-86d8-c833276deaec",
      "status": "Failed",
      "startTime": "2026-09-24T07:41:08.321205783Z",
      "endTime": "2026-09-24T08:10:12.392342627Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000fced80>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-tmsknb268d6w" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/03151659-c9c2-4800-86d8-c833276deaec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/03151659-c9c2-4800-86d8-c833276deaec",
      "name": "03151659-c9c2-4800-86d8-c833276deaec",
      "status": "Failed",
      "startTime": "2026-09-24T07:41:08.321205783Z",
      "endTime": "2026-09-24T08:10:12.392342627Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-kzgnsh" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000be2600>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-kzgnsh: failed waiting for cluster="cp-ystream-upgrade-5-0-kzgnsh" in resourcegroup="rg-cp-ystream-upgrade-5-0-nh8nv9nrhbfl" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e49bd5a2-59c3-457f-8f93-e961de30f26b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e49bd5a2-59c3-457f-8f93-e961de30f26b",
      "name": "e49bd5a2-59c3-457f-8f93-e961de30f26b",
      "status": "Failed",
      "startTime": "2026-09-24T07:41:06.715026226Z",
      "endTime": "2026-09-24T08:10:12.435086078Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-kzgnsh" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000be2600>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-kzgnsh: failed waiting for cluster="cp-ystream-upgrade-5-0-kzgnsh" in resourcegroup="rg-cp-ystream-upgrade-5-0-nh8nv9nrhbfl" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e49bd5a2-59c3-457f-8f93-e961de30f26b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e49bd5a2-59c3-457f-8f93-e961de30f26b",
      "name": "e49bd5a2-59c3-457f-8f93-e961de30f26b",
      "status": "Failed",
      "startTime": "2026-09-24T07:41:06.715026226Z",
      "endTime": "2026-09-24T08:10:12.435086078Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (45)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:52	PR #6503	2103014767818969088

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T01:36:59Z
  Ended:       2026-09-24T01:43:01Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8079616-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8079616/providers/microsoft.monitor/accounts/hcps-j8079616", namespace="ocm-arohcpci01-2t12nvqr38rk88v2oib5stvkqbcsin5h-r3r9m3h6q3b6v7a", pod="arohcp-monitor-kube-state-metrics-7955558448-q4z8l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t12nvqr38rk88v2oib5stvkqbcsin5h-r3r9m3h6q3b6v7a/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-24T01:36:59Z
  Ended:       2026-09-24T01:43:01Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8079616-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8079616/providers/microsoft.monitor/accounts/hcps-j8079616", namespace="ocm-arohcpci01-2t12o9jo7duvi4mjl85qtghmc0bu27i9-j6r7x7q1c9p4g7n", pod="arohcp-monitor-kube-state-metrics-7955558448-q4z8l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t12o9jo7duvi4mjl85qtghmc0bu27i9-j6r7x7q1c9p4g7n/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-24T01:37:00Z
  Ended:       2026-09-24T01:43:01Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8079616-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8079616/providers/microsoft.monitor/accounts/hcps-j8079616", namespace="ocm-arohcpci01-2t12nvj5ubvth7im4d53igd84r482cjj-n2f1o7y4s2y9x7u", pod="arohcp-monitor-kube-state-metrics-7955558448-q4z8l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t12nvj5ubvth7im4d53igd84r482cjj-n2f1o7y4s2y9x7u/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-24T01:37:00Z
  Ended:       2026-09-24T01:43:01Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8079616-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8079616/providers/microsoft.monitor/accounts/hcps-j8079616", namespace="ocm-arohcpci01-2t12o9jo7duvi4mjl85qtghmc0bu27i9-j6r7x7q1c9p4g7n", pod="arohcp-monitor-kube-state-metrics-7955558448-q4z8l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t12o9jo7duvi4mjl85qtghmc0bu27i9-j6r7x7q1c9p4g7n/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 00:35	PR #6917	2102919814468079616

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T08:07:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-7jtls", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-24T08:19:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-7jtls", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-24T08:19:22Z
  Ended:       2026-09-24T08:37:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="klusterlet-2t18bqdbena2h0p1tg4m0lfs0js71a72", pod="arohcp-monitor-kube-state-metrics-7955558448-7jtls", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t18bqdbena2h0p1tg4m0lfs0js71a72/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:52	PR #6503	2103014767818969088

alert [svc] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [svc] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T08:19:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-7jtls", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-24T08:23:25Z
  Ended:       2026-09-24T08:37:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="klusterlet-2t18bqdbena2h0p1tg4m0lfs0js71a72", pod="arohcp-monitor-kube-state-metrics-7955558448-7jtls", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t18bqdbena2h0p1tg4m0lfs0js71a72/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-24T08:44:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-7jtls", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:52	PR #6503	2103014767818969088

cluster entered terminal Failed state after role assignment deployment

full failure pattern:

cluster entered terminal Failed state after role assignment deployment

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:52	PR #6503	2103014767818969088

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator, openshift-apiserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc0005d4e40>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-mdj8lgwtvhhv" to finish creating: GET https://rp.j8049152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1e1d081b-a194-4d7e-ad25-c5f0f75190c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1e1d081b-a194-4d7e-ad25-c5f0f75190c7",
      "name": "1e1d081b-a194-4d7e-ad25-c5f0f75190c7",
      "status": "Failed",
      "startTime": "2026-09-24T12:47:20.580135273Z",
      "endTime": "2026-09-24T13:16:21.048177334Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 1m5s ago; version 4.21.34 is Partial (want Completed), started 3m10s ago; version 4.21.34 is Partial (want Completed), started 6m2s ago; version 4.21.34 is Partial (want Completed), started 6m38s ago; version 4.21.34 is Partial (want Completed), started 10m34s ago; version 4.21.34 is Partial (want Completed), started 16m16s ago; version 4.21.34 is Partial (want Completed), started 20m17s ago; version 4.21.34 is Partial (want Completed), started 23m52s ago; version 4.21.34 is Partial (want Completed), started 24m53s ago; version 4.21.34 is Partial (want Completed), started 25m36s ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc0005d4e40>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-mdj8lgwtvhhv" to finish creating: GET https://rp.j8049152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1e1d081b-a194-4d7e-ad25-c5f0f75190c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1e1d081b-a194-4d7e-ad25-c5f0f75190c7",
      "name": "1e1d081b-a194-4d7e-ad25-c5f0f75190c7",
      "status": "Failed",
      "startTime": "2026-09-24T12:47:20.580135273Z",
      "endTime": "2026-09-24T13:16:21.048177334Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 1m5s ago; version 4.21.34 is Partial (want Completed), started 3m10s ago; version 4.21.34 is Partial (want Completed), started 6m2s ago; version 4.21.34 is Partial (want Completed), started 6m38s ago; version 4.21.34 is Partial (want Completed), started 10m34s ago; version 4.21.34 is Partial (want Completed), started 16m16s ago; version 4.21.34 is Partial (want Completed), started 20m17s ago; version 4.21.34 is Partial (want Completed), started 23m52s ago; version 4.21.34 is Partial (want Completed), started 24m53s ago; version 4.21.34 is Partial (want Completed), started 25m36s ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 11:47	PR #7123	2103089013568049152

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machine...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2nj474" for cluster "cluster-candidate-5-0-2nj474"
Unexpected error:
    <*fmt.wrapError | 0xc000b7a380>: 
    failed to create NodePool np-2nj474: failed waiting for nodepool="np-2nj474" for cluster "cluster-candidate-5-0-2nj474" in resourcegroup="rg-candidate-5-0-vjxqrms5jclz" to finish creating: GET https://rp.j9877248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/514bc0c8-a828-4772-95da-f95954e05701
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/514bc0c8-a828-4772-95da-f95954e05701",
      "name": "514bc0c8-a828-4772-95da-f95954e05701",
      "status": "Failed",
      "startTime": "2026-09-24T23:48:54.785691346Z",
      "endTime": "2026-09-25T00:08:00.18158216Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np to exist\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np to exist\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2nj474" for cluster "cluster-candidate-5-0-2nj474"
Unexpected error:
    <*fmt.wrapError | 0xc000b7a380>: 
    failed to create NodePool np-2nj474: failed waiting for nodepool="np-2nj474" for cluster "cluster-candidate-5-0-2nj474" in resourcegroup="rg-candidate-5-0-vjxqrms5jclz" to finish creating: GET https://rp.j9877248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/514bc0c8-a828-4772-95da-f95954e05701
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/514bc0c8-a828-4772-95da-f95954e05701",
      "name": "514bc0c8-a828-4772-95da-f95954e05701",
      "status": "Failed",
      "startTime": "2026-09-24T23:48:54.785691346Z",
      "endTime": "2026-09-25T00:08:00.18158216Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np to exist\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-s79np to exist\nMachine m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-vjxqrms5jclz-candidate-2nj474--managed/providers/Microsoft.Compute/virtualMachines/m6g6a2k9x2n9s0x-np-2nj474-2qbdc-tcv66 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 22:38	PR #7123	2103252727629877248

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machine...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000cea980>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dx85jrjs9gbf" to finish creating: GET https://rp.j7862784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ddf0b59-a1c5-4f88-9e2a-09c2b8b441c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ddf0b59-a1c5-4f88-9e2a-09c2b8b441c9",
      "name": "9ddf0b59-a1c5-4f88-9e2a-09c2b8b441c9",
      "status": "Failed",
      "startTime": "2026-09-24T17:42:43.803875728Z",
      "endTime": "2026-09-24T18:01:44.201124036Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dx85jrjs9gbf--managed/providers/Microsoft.Compute/virtualMachines/j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dx85jrjs9gbf--managed/providers/Microsoft.Compute/virtualMachines/j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000cea980>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dx85jrjs9gbf" to finish creating: GET https://rp.j7862784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ddf0b59-a1c5-4f88-9e2a-09c2b8b441c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ddf0b59-a1c5-4f88-9e2a-09c2b8b441c9",
      "name": "9ddf0b59-a1c5-4f88-9e2a-09c2b8b441c9",
      "status": "Failed",
      "startTime": "2026-09-24T17:42:43.803875728Z",
      "endTime": "2026-09-24T18:01:44.201124036Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dx85jrjs9gbf--managed/providers/Microsoft.Compute/virtualMachines/j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-dx85jrjs9gbf--managed/providers/Microsoft.Compute/virtualMachines/j3g0e5m6b3n7n1m-np-1-45jtt-ld7k7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 16:41	PR #6821	2103162932127862784

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no s...

full failure pattern:

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:209]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00268e060>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-dr7hw.apps.aro.cilium-cluster.vzkq.j7691264.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dr7hw.apps.aro.cilium-cluster.vzkq.j7691264.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dr7hw.apps.aro.cilium-cluster.vzkq.j7691264.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:209]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00268e060>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-dr7hw.apps.aro.cilium-cluster.vzkq.j7691264.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dr7hw.apps.aro.cilium-cluster.vzkq.j7691264.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dr7hw.apps.aro.cilium-cluster.vzkq.j7691264.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 19:09	PR #6821	2103200143447691264

alert [svc] KubeContainerOOMKilled fired

full failure pattern:

alert [svc] KubeContainerOOMKilled fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T08:49:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", container="prometheus-collector", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="kube-system", pod="ama-metrics-node-5rpz7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="OOMKilled", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8afb0369-75a6-47c9-a399-fdc0457fc23d"
  Description: Container prometheus-collector in pod kube-system/ama-metrics-node-5rpz7 on cluster ci01-j8969088-mgmt-1 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-24T08:50:13Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j8969088-mgmt-1", component="kubernetes-infrastructure", container="prometheus-collector", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8969088/providers/microsoft.monitor/accounts/services-j8969088", namespace="kube-system", pod="ama-metrics-node-5rpz7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="OOMKilled", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8afb0369-75a6-47c9-a399-fdc0457fc23d"
  Description: Container prometheus-collector in pod kube-system/ama-metrics-node-5rpz7 on cluster ci01-j8969088-mgmt-1 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeContainerOOMKilled does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:52	PR #6503	2103014767818969088

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j6380032-mgmt-2","clusterName":"ci01-j638003

Signal:

provision

none

Full failure examples (1)

time=2026-09-24T19:21:37.035Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-24T19:51:36.670Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-24T19:21:37.041921702Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6380032-mgmt-2\",\"clusterName\":\"ci01-j6380032-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-24T19:21:38.603615087Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6380032-mgmt-2\",\"clusterName\":\"ci01-j6380032-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-24T19:21:39.051190133Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6380032-mgmt-2\",\"clusterName\":\"ci01-j6380032-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 18:40	PR #7045	2103192859166380032

failed to query public IP echo service: Get "<url>": net/http: TLS handshake timeout

full failure pattern:

failed to query public IP echo service: Get "<url>": net/http: TLS handshake timeout

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:172]: failed to resolve test runner public IP
Unexpected error:
    <*fmt.wrapError | 0xc000d70460>: 
    failed to query public IP echo service: Get "https://checkip.amazonaws.com": net/http: TLS handshake timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:172]: failed to resolve test runner public IP
Unexpected error:
    <*fmt.wrapError | 0xc000d70460>: 
    failed to query public IP echo service: Get "https://checkip.amazonaws.com": net/http: TLS handshake timeout
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 22:18	PR #7148	2103247783359156224

timeout during CreateHCPClusterAndWait; context deadline exceeded

full failure pattern:

timeout during CreateHCPClusterAndWait; context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000f4e930>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-mkjrz7qtzs5m" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-mkjrz7qtzs5m, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000f4e930>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-mkjrz7qtzs5m" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-mkjrz7qtzs5m, error: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:52	PR #6503	2103014767818969088

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000fec560>: 
    DNS for route host agnhost-e2e-sample-app-f5rnl.apps.aro.v20260901.t5ah.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f5rnl.apps.aro.v20260901.t5ah.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f5rnl.apps.aro.v20260901.t5ah.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000fec560>: 
    DNS for route host agnhost-e2e-sample-app-f5rnl.apps.aro.v20260901.t5ah.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f5rnl.apps.aro.v20260901.t5ah.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f5rnl.apps.aro.v20260901.t5ah.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:44	PR #6915	2103012742733500416

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-bxncxk"
Unexpected error:
    <*fmt.wrapError | 0xc000b16880>: 
    DNS for route host agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-bxncxk"
Unexpected error:
    <*fmt.wrapError | 0xc000b16880>: 
    DNS for route host agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 06:44	PR #6915	2103012742733500416

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc002fcb410>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-jgw9qgklj7f5" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-jgw9qgklj7f5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc002fcb410>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-jgw9qgklj7f5" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-jgw9qgklj7f5, error: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 18:40	PR #7135	2103192892754366464

alert [svc] ArobitForwarderJobUp fired

full failure pattern:

alert [svc] ArobitForwarderJobUp fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:33:47Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j4604416-svc", component="monitoring-infrastructure", region="westus3", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j4604416-svc has been unreachable for 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ArobitForwarderJobUp does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 16:34	PR #6820	2103161107324604416

alert [svc] BackendControllerRetryHotLoop fired

full failure pattern:

alert [svc] BackendControllerRetryHotLoop fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T15:19:32Z
  Ended:       2026-09-24T16:14:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j2833536-svc", component="backend", name="controlplaneversionstatuscollector", region="westus3", severity="warning"
  Description: Backend controller workqueue controlplaneversionstatuscollector has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 14:18	PR #6577	2103126923872833536

alert [svc] KubeDaemonSetRolloutStuck fired

full failure pattern:

alert [svc] KubeDaemonSetRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:34:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.41:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4604416/providers/microsoft.monitor/accounts/services-j4604416", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-mmg7d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:34:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.41:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4604416/providers/microsoft.monitor/accounts/services-j4604416", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-mmg7d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 16:34	PR #6820	2103161107324604416

Environment: INT

Failure patterns: 1 · Failures matched: 1 · Total runs: 6 · Runs affected: 1

Failure pattern

Failed at

Trend

Also in

VerifyBasicAccess failed: failed to list services: Get "<url>

full failure pattern:

VerifyBasicAccess failed: failed to list services: Get "<url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:155]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc00158c4c8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/api/v1/namespaces/default/services": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:155]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc00158c4c8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/api/v1/namespaces/default/services": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 14:57	n/a	2103136634483511296

Environment: PROD

Failure patterns: 10 · Failures matched: 28 · Total runs: 5 · Runs affected: 5

Failure pattern

Failed at

Trend

Also in

storage account log verification failed

full failure pattern:

storage account log verification failed

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxjdxchm and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-lthncjjflrt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc001150fb0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxjdxchm and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-lthncjjflrt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc001150fb0>: 
    storage account log verification failed
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxdxtlkr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-zhq2pjd4clbv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc00066d200>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxdxtlkr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-zhq2pjd4clbv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc00066d200>: 
    storage account log verification failed
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxj5dtdr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-v7bfrx7ddjpw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000e01080>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxj5dtdr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-v7bfrx7ddjpw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000e01080>: 
    storage account log verification failed
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:38	n/a	2103267802377883648
2026-09-24 23:00	n/a	2103258234201575424
2026-09-24 20:45	n/a	2103224303930576896
2026-09-24 17:14	n/a	2103171230415720448

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

DEV, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00088a9a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-6bxr9l" in resourcegroup="rg-np-noedge-6bxr9l-s8q7zdxcxq2b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2b9dc29e-fe8f-45b4-b8b8-139dfe59a7f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2b9dc29e-fe8f-45b4-b8b8-139dfe59a7f6",
      "name": "2b9dc29e-fe8f-45b4-b8b8-139dfe59a7f6",
      "status": "Failed",
      "startTime": "2026-09-25T01:01:50.31869719Z",
      "endTime": "2026-09-25T01:20:50.832464334Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00088a9a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-6bxr9l" in resourcegroup="rg-np-noedge-6bxr9l-s8q7zdxcxq2b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2b9dc29e-fe8f-45b4-b8b8-139dfe59a7f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2b9dc29e-fe8f-45b4-b8b8-139dfe59a7f6",
      "name": "2b9dc29e-fe8f-45b4-b8b8-139dfe59a7f6",
      "status": "Failed",
      "startTime": "2026-09-25T01:01:50.31869719Z",
      "endTime": "2026-09-25T01:20:50.832464334Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000b24be0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-qvrt9xzb9h7m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/05a99209-7724-468c-8d11-789afc14d5d3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/05a99209-7724-468c-8d11-789afc14d5d3",
      "name": "05a99209-7724-468c-8d11-789afc14d5d3",
      "status": "Failed",
      "startTime": "2026-09-24T23:58:35.632366587Z",
      "endTime": "2026-09-25T00:17:38.767686221Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000b24be0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-qvrt9xzb9h7m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/05a99209-7724-468c-8d11-789afc14d5d3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/05a99209-7724-468c-8d11-789afc14d5d3",
      "name": "05a99209-7724-468c-8d11-789afc14d5d3",
      "status": "Failed",
      "startTime": "2026-09-24T23:58:35.632366587Z",
      "endTime": "2026-09-25T00:17:38.767686221Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0011137a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-vhxnnlnn4h55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8121b572-5dd9-49c7-9e1e-bb37581297c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8121b572-5dd9-49c7-9e1e-bb37581297c6",
      "name": "8121b572-5dd9-49c7-9e1e-bb37581297c6",
      "status": "Failed",
      "startTime": "2026-09-24T20:58:20.871989951Z",
      "endTime": "2026-09-24T21:17:23.599073745Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0011137a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-vhxnnlnn4h55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8121b572-5dd9-49c7-9e1e-bb37581297c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8121b572-5dd9-49c7-9e1e-bb37581297c6",
      "name": "8121b572-5dd9-49c7-9e1e-bb37581297c6",
      "status": "Failed",
      "startTime": "2026-09-24T20:58:20.871989951Z",
      "endTime": "2026-09-24T21:17:23.599073745Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:38	n/a	2103267802377883648
2026-09-24 23:00	n/a	2103258234201575424
2026-09-24 20:45	n/a	2103224303930576896

ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000ab7ea0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-r6szljgv5m4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-r6szljgv5m4p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000ab7ea0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-r6szljgv5m4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-r6szljgv5m4p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000e90d80>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-pc4xbw2wvdgz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-pc4xbw2wvdgz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000e90d80>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-pc4xbw2wvdgz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-pc4xbw2wvdgz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0015fdec0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-gk9hkpv4f4bq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0015fdec0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-gk9hkpv4f4bq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (7)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 02:00	n/a	2102941141266599936

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0017055c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-vj7bqc7l746k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0017055c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-vj7bqc7l746k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0002b1230>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cp45sss7fz6v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0002b1230>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cp45sss7fz6v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc0010a2930>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-grftwltzcrgn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc0010a2930>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-grftwltzcrgn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234201575424

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc0010c9720>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-r2rlkr" resourcegroup="rg-np-dg-minor-r2rlkr-ngfwrz48zjxx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-r2rlkr-ngfwrz48zjxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-r2rlkr/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc0010c9720>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-r2rlkr" resourcegroup="rg-np-dg-minor-r2rlkr-ngfwrz48zjxx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-r2rlkr-ngfwrz48zjxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-r2rlkr/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-d25jl2" for cluster "cluster-nightly-4-20-d25jl2"
Unexpected error:
    <*fmt.wrapError | 0xc000f0a140>: 
    failed to create NodePool np-d25jl2: failed starting nodepool creation "np-d25jl2" for cluster "cluster-nightly-4-20-d25jl2" in resourcegroup="rg-nightly-4-20-kl7rft6vttsr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kl7rft6vttsr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-d25jl2/nodePools/np-d25jl2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-d25jl2" for cluster "cluster-nightly-4-20-d25jl2"
Unexpected error:
    <*fmt.wrapError | 0xc000f0a140>: 
    failed to create NodePool np-d25jl2: failed starting nodepool creation "np-d25jl2" for cluster "cluster-nightly-4-20-d25jl2" in resourcegroup="rg-nightly-4-20-kl7rft6vttsr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kl7rft6vttsr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-d25jl2/nodePools/np-d25jl2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000d9eba0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ftgp6q" in resourcegroup="rg-np-version-upgrade-ftgp6q-8slz9vtsg7dc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ftgp6q-8slz9vtsg7dc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ftgp6q/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000d9eba0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ftgp6q" in resourcegroup="rg-np-version-upgrade-ftgp6q-8slz9vtsg7dc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ftgp6q-8slz9vtsg7dc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ftgp6q/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234201575424

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

STG

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c772d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c772d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234201575424

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster servi...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-854hq8s8lwp6/cluster-nightly-5-1-npws6l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ed0ea0>: 
    failed to create HCP cluster cluster-nightly-5-1-npws6l: failed waiting for cluster="cluster-nightly-5-1-npws6l" in resourcegroup="rg-nightly-5-1-854hq8s8lwp6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5427e4d8-cab2-43c9-ac23-7d670008e23b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5427e4d8-cab2-43c9-ac23-7d670008e23b",
      "name": "5427e4d8-cab2-43c9-ac23-7d670008e23b",
      "status": "Failed",
      "startTime": "2026-09-24T23:33:11.266527055Z",
      "endTime": "2026-09-25T00:02:18.550348976Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-854hq8s8lwp6/cluster-nightly-5-1-npws6l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ed0ea0>: 
    failed to create HCP cluster cluster-nightly-5-1-npws6l: failed waiting for cluster="cluster-nightly-5-1-npws6l" in resourcegroup="rg-nightly-5-1-854hq8s8lwp6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5427e4d8-cab2-43c9-ac23-7d670008e23b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5427e4d8-cab2-43c9-ac23-7d670008e23b",
      "name": "5427e4d8-cab2-43c9-ac23-7d670008e23b",
      "status": "Failed",
      "startTime": "2026-09-24T23:33:11.266527055Z",
      "endTime": "2026-09-25T00:02:18.550348976Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234201575424

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-ml64wq with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0013c02a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-ml64wq: failed waiting for cluster="np-version-upgrade-cluster-ml64wq" in resourcegroup="rg-np-version-upgrade-ml64wq-zkbzk7hvlvck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "name": "2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "status": "Failed",
      "startTime": "2026-09-25T00:48:56.872501334Z",
      "endTime": "2026-09-25T01:18:00.947451943Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-ml64wq with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0013c02a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-ml64wq: failed waiting for cluster="np-version-upgrade-cluster-ml64wq" in resourcegroup="rg-np-version-upgrade-ml64wq-zkbzk7hvlvck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "name": "2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "status": "Failed",
      "startTime": "2026-09-25T00:48:56.872501334Z",
      "endTime": "2026-09-25T01:18:00.947451943Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:38	n/a	2103267802377883648

Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again

full failure pattern:

Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x
Unexpected error:
    <*errors.StatusError | 0xc00103ab40>: 
    Operation cannot be fulfilled on nodes "a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x
Unexpected error:
    <*errors.StatusError | 0xc00103ab40>: 
    Operation cannot be fulfilled on nodes "a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 17:14	n/a	2103171230415720448

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001321160>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-l2xrb7ls9tkm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e3486a0-a8be-491f-b7ba-de40bdc6a4c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e3486a0-a8be-491f-b7ba-de40bdc6a4c5",
      "name": "4e3486a0-a8be-491f-b7ba-de40bdc6a4c5",
      "status": "Failed",
      "startTime": "2026-09-24T03:02:54.530721397Z",
      "endTime": "2026-09-24T03:21:54.293711104Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001321160>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-l2xrb7ls9tkm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e3486a0-a8be-491f-b7ba-de40bdc6a4c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e3486a0-a8be-491f-b7ba-de40bdc6a4c5",
      "name": "4e3486a0-a8be-491f-b7ba-de40bdc6a4c5",
      "status": "Failed",
      "startTime": "2026-09-24T03:02:54.530721397Z",
      "endTime": "2026-09-24T03:21:54.293711104Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to delete a nodepool	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 02:00	n/a	2102941141266599936

Environment: STG

Failure patterns: 11 · Failures matched: 420 · Total runs: 8 · Runs affected: 8

Failure pattern

Failed at

Trend

Also in

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHat...

full failure pattern:

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-p49qttrgxjfs/cluster-candidate-4-23-h8xv9w should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a0e9e0>: 
    failed to create HCP cluster cluster-candidate-4-23-h8xv9w: failed waiting for cluster="cluster-candidate-4-23-h8xv9w" in resourcegroup="rg-candidate-4-23-p49qttrgxjfs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/775da561-b2d7-4a7a-bbc2-e2713e469817
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-p49qttrgxjfs/cluster-candidate-4-23-h8xv9w should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a0e9e0>: 
    failed to create HCP cluster cluster-candidate-4-23-h8xv9w: failed waiting for cluster="cluster-candidate-4-23-h8xv9w" in resourcegroup="rg-candidate-4-23-p49qttrgxjfs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/775da561-b2d7-4a7a-bbc2-e2713e469817
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-q9n7xf" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0015a82e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-q9n7xf: failed waiting for cluster="cp-ystream-upgrade-5-0-q9n7xf" in resourcegroup="rg-cp-ystream-upgrade-5-0-mbh76rk9sprx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e50bc50d-ed3a-44e5-99bf-5d82436a9665
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-q9n7xf" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0015a82e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-q9n7xf: failed waiting for cluster="cp-ystream-upgrade-5-0-q9n7xf" in resourcegroup="rg-cp-ystream-upgrade-5-0-mbh76rk9sprx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e50bc50d-ed3a-44e5-99bf-5d82436a9665
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000994380>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2v5hnvkbp6nz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ac014577-36dc-4208-bce8-f0e66ebed0a8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000994380>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2v5hnvkbp6nz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ac014577-36dc-4208-bce8-f0e66ebed0a8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (61)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	5
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to list available HCP OpenShift versions and validate response content	5
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to delete a nodepool	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to delete an HCP cluster whose managed identities were deleted first	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to update node pool labels and taints	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	3
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to list available HCP OpenShift versions and validate response content	1

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-24 13:55	n/a	2103121232214888448
2026-09-24 09:46	n/a	2103058579819859968
2026-09-24 08:37	n/a	2103041133276827648
2026-09-24 05:07	n/a	2102988159166976000
2026-09-24 04:47	n/a	2102983195996721152
2026-09-24 02:00	n/a	2102941141295960064

ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.

full failure pattern:

ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:73]: failed to create customer resources for v20261001preview cluster
Unexpected error:
    <*fmt.wrapError | 0xc00106f760>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-v20261001-x6gm79" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-v20261001-x6gm79/operationStatuses/08584113797462734569
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-w8qdl2mz4kg9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-7e767rdphst3w\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-12/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:73]: failed to create customer resources for v20261001preview cluster
Unexpected error:
    <*fmt.wrapError | 0xc00106f760>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-v20261001-x6gm79" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-v20261001-x6gm79/operationStatuses/08584113797462734569
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-w8qdl2mz4kg9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-7e767rdphst3w\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-12/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:114]: failed to create customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b13100>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ingress-cert-le-5tmv7b" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ingress-cert-le-5tmv7b/operationStatuses/08584113797516953541
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-qzpjtfdnhg6c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-nhimtfs3fueps\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:114]: failed to create customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b13100>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ingress-cert-le-5tmv7b" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ingress-cert-le-5tmv7b/operationStatuses/08584113797516953541
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-qzpjtfdnhg6c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-nhimtfs3fueps\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:283]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000e7dae0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-4lkc6c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-4lkc6c/operationStatuses/08584113794966440147
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-48zl586s8mmx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:283]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000e7dae0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-4lkc6c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-4lkc6c/operationStatuses/08584113794966440147
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-48zl586s8mmx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (113)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to delete a nodepool	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to delete an HCP cluster whose managed identities were deleted first	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to update node pool labels and taints	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to delete an HCP cluster whose managed identities were deleted first	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-24 05:07	n/a	2102988159166976000
2026-09-24 04:47	n/a	2102983195996721152
2026-09-24 02:00	n/a	2102941141295960064

ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'...

full failure pattern:

ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000b7e680>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-d5k7vz9fgmp5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-d5k7vz9fgmp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000b7e680>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-d5k7vz9fgmp5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-d5k7vz9fgmp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-5zxmn6" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e7afc0>: 
    failed to create HCP cluster cluster-zstream-4-20-5zxmn6: failed starting cluster creation "cluster-zstream-4-20-5zxmn6" in resourcegroup="rg-zstream-upgrade-4-20-87qdz9hglgx4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-87qdz9hglgx4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-5zxmn6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-5zxmn6" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e7afc0>: 
    failed to create HCP cluster cluster-zstream-4-20-5zxmn6: failed starting cluster creation "cluster-zstream-4-20-5zxmn6" in resourcegroup="rg-zstream-upgrade-4-20-87qdz9hglgx4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-87qdz9hglgx4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-5zxmn6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-25fcqw
Unexpected error:
    <*fmt.wrapError | 0xc000f12e80>: 
    failed to create HCP cluster np-noedge-25fcqw: failed starting cluster creation "np-noedge-25fcqw" in resourcegroup="rg-np-noedge-25fcqw-mkk44mbtzfcl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-25fcqw-mkk44mbtzfcl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-25fcqw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-25fcqw
Unexpected error:
    <*fmt.wrapError | 0xc000f12e80>: 
    failed to create HCP cluster np-noedge-25fcqw: failed starting cluster creation "np-noedge-25fcqw" in resourcegroup="rg-np-noedge-25fcqw-mkk44mbtzfcl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-25fcqw-mkk44mbtzfcl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-25fcqw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (58)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to delete an HCP cluster whose managed identities were deleted first	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 20:07	n/a	2103214845615673344

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0019711d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7dp4spk72w92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0019711d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7dp4spk72w92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001566b10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z76xpv9lv7mg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001566b10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z76xpv9lv7mg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0013a3320>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cgk7n47ncfl4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0013a3320>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cgk7n47ncfl4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234264489985

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000f5c1c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-k5htqv" in resourcegroup="rg-np-skip-minor-k5htqv-87tsfgngkbxz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-k5htqv-87tsfgngkbxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-k5htqv/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000f5c1c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-k5htqv" in resourcegroup="rg-np-skip-minor-k5htqv-87tsfgngkbxz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-k5htqv-87tsfgngkbxz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-k5htqv/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-g6wkqd" for cluster "cluster-nightly-4-20-g6wkqd"
Unexpected error:
    <*fmt.wrapError | 0xc000926600>: 
    failed to create NodePool np-g6wkqd: failed starting nodepool creation "np-g6wkqd" for cluster "cluster-nightly-4-20-g6wkqd" in resourcegroup="rg-nightly-4-20-2j2bp6ng4gdp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2j2bp6ng4gdp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-g6wkqd/nodePools/np-g6wkqd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-g6wkqd" for cluster "cluster-nightly-4-20-g6wkqd"
Unexpected error:
    <*fmt.wrapError | 0xc000926600>: 
    failed to create NodePool np-g6wkqd: failed starting nodepool creation "np-g6wkqd" for cluster "cluster-nightly-4-20-g6wkqd" in resourcegroup="rg-nightly-4-20-2j2bp6ng4gdp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2j2bp6ng4gdp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-g6wkqd/nodePools/np-g6wkqd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc00151a140>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4pqnxp" resourcegroup="rg-np-dg-minor-4pqnxp-4lnmhzl52tvp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4pqnxp-4lnmhzl52tvp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4pqnxp/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc00151a140>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4pqnxp" resourcegroup="rg-np-dg-minor-4pqnxp-4lnmhzl52tvp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4pqnxp-4lnmhzl52tvp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4pqnxp/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234264489985

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

DEV, PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000d07ae0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-xxwmzws6kxxc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "name": "85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "status": "Failed",
      "startTime": "2026-09-24T23:13:56.752561651Z",
      "endTime": "2026-09-24T23:32:59.664565124Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000d07ae0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-xxwmzws6kxxc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "name": "85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "status": "Failed",
      "startTime": "2026-09-24T23:13:56.752561651Z",
      "endTime": "2026-09-24T23:32:59.664565124Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and custom node pool osDisk size	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234264489985

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016823c0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016823c0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234264489985

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-wllchtq6tspw/cluster-nightly-5-1-xqrkdk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000614980>: 
    failed to create HCP cluster cluster-nightly-5-1-xqrkdk: failed waiting for cluster="cluster-nightly-5-1-xqrkdk" in resourcegroup="rg-nightly-5-1-wllchtq6tspw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "name": "8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "status": "Failed",
      "startTime": "2026-09-24T23:29:29.214562248Z",
      "endTime": "2026-09-24T23:58:29.739929622Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-wllchtq6tspw/cluster-nightly-5-1-xqrkdk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000614980>: 
    failed to create HCP cluster cluster-nightly-5-1-xqrkdk: failed waiting for cluster="cluster-nightly-5-1-xqrkdk" in resourcegroup="rg-nightly-5-1-wllchtq6tspw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "name": "8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "status": "Failed",
      "startTime": "2026-09-24T23:29:29.214562248Z",
      "endTime": "2026-09-24T23:58:29.739929622Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234264489985

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cf4l7z with version 4.21.0-0.nightly-multi-2026-09-24-033657
Unexpected error:
    <*fmt.wrapError | 0xc00142c760>: 
    failed to create HCP cluster np-version-upgrade-cluster-cf4l7z: failed waiting for cluster="np-version-upgrade-cluster-cf4l7z" in resourcegroup="rg-np-version-upgrade-cf4l7z-c8gcs9n4rck7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "name": "58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "status": "Failed",
      "startTime": "2026-09-24T23:42:40.568397466Z",
      "endTime": "2026-09-25T00:11:49.734192333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cf4l7z with version 4.21.0-0.nightly-multi-2026-09-24-033657
Unexpected error:
    <*fmt.wrapError | 0xc00142c760>: 
    failed to create HCP cluster np-version-upgrade-cluster-cf4l7z: failed waiting for cluster="np-version-upgrade-cluster-cf4l7z" in resourcegroup="rg-np-version-upgrade-cf4l7z-c8gcs9n4rck7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "name": "58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "status": "Failed",
      "startTime": "2026-09-24T23:42:40.568397466Z",
      "endTime": "2026-09-25T00:11:49.734192333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234264489985

ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpen...

full failure pattern:

ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, uksouth'.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0017589f0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: NoRegisteredProviderFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "NoRegisteredProviderFound",
        "message": "No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0017589f0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: NoRegisteredProviderFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "NoRegisteredProviderFound",
        "message": "No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to list available HCP OpenShift versions and validate response content	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 20:07	n/a	2103214845615673344

storage account log verification failed

full failure pattern:

storage account log verification failed

Signal:

e2e

PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox6npjq8 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-g4qtcfdblkfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000d18d20>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox6npjq8 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-g4qtcfdblkfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000d18d20>: 
    storage account log verification failed
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-24 23:00	n/a	2103258234264489985