CIHealth Failure Patterns

Environment: DEV

Failure patterns: 54 · Failures matched: 209 · Total runs: 149 · Runs affected: 108

Failure pattern

Failed at

Trend

Also in

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator, openshift-apiserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000c1a600>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-sm4jtnxclsxs" to finish creating: GET https://rp.j5868928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d34f713-a16b-433b-99d2-1c7b278e6b21
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d34f713-a16b-433b-99d2-1c7b278e6b21",
      "name": "6d34f713-a16b-433b-99d2-1c7b278e6b21",
      "status": "Failed",
      "startTime": "2026-09-25T19:01:13.53147986Z",
      "endTime": "2026-09-25T19:30:14.113086545Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 3m12s ago; version 4.21.35 is Partial (want Completed), started 4m17s ago; version 4.21.35 is Partial (want Completed), started 5m48s ago; version 4.21.35 is Partial (want Completed), started 13m55s ago; version 4.21.35 is Partial (want Completed), started 15m34s ago; version 4.21.35 is Partial (want Completed), started 19m39s ago; version 4.21.35 is Partial (want Completed), started 23m46s ago; version 4.21.35 is Partial (want Completed), started 24m59s ago; version 4.21.35 is Partial (want Completed), started 25m33s ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000c1a600>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-sm4jtnxclsxs" to finish creating: GET https://rp.j5868928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d34f713-a16b-433b-99d2-1c7b278e6b21
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d34f713-a16b-433b-99d2-1c7b278e6b21",
      "name": "6d34f713-a16b-433b-99d2-1c7b278e6b21",
      "status": "Failed",
      "startTime": "2026-09-25T19:01:13.53147986Z",
      "endTime": "2026-09-25T19:30:14.113086545Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 3m12s ago; version 4.21.35 is Partial (want Completed), started 4m17s ago; version 4.21.35 is Partial (want Completed), started 5m48s ago; version 4.21.35 is Partial (want Completed), started 13m55s ago; version 4.21.35 is Partial (want Completed), started 15m34s ago; version 4.21.35 is Partial (want Completed), started 19m39s ago; version 4.21.35 is Partial (want Completed), started 23m46s ago; version 4.21.35 is Partial (want Completed), started 24m59s ago; version 4.21.35 is Partial (want Completed), started 25m33s ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 18:09	PR #7178	2103547336255868928

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [openshift-apiserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f4e300>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-k7vshcl5nh9s" to finish creating: GET https://rp.j8937088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/60dc2458-30c4-43c0-9c09-17e7a502b57d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/60dc2458-30c4-43c0-9c09-17e7a502b57d",
      "name": "60dc2458-30c4-43c0-9c09-17e7a502b57d",
      "status": "Failed",
      "startTime": "2026-09-25T17:21:27.919326113Z",
      "endTime": "2026-09-25T17:50:31.48468713Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 35s ago; version 4.21.35 is Partial (want Completed), started 3m12s ago; version 4.21.35 is Partial (want Completed), started 5m19s ago; version 4.21.35 is Partial (want Completed), started 6m43s ago; version 4.21.35 is Partial (want Completed), started 8m41s ago; version 4.21.35 is Partial (want Completed), started 12m14s ago; version 4.21.35 is Partial (want Completed), started 13m25s ago; version 4.21.35 is Partial (want Completed), started 13m48s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f4e300>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-k7vshcl5nh9s" to finish creating: GET https://rp.j8937088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/60dc2458-30c4-43c0-9c09-17e7a502b57d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/60dc2458-30c4-43c0-9c09-17e7a502b57d",
      "name": "60dc2458-30c4-43c0-9c09-17e7a502b57d",
      "status": "Failed",
      "startTime": "2026-09-25T17:21:27.919326113Z",
      "endTime": "2026-09-25T17:50:31.48468713Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 35s ago; version 4.21.35 is Partial (want Completed), started 3m12s ago; version 4.21.35 is Partial (want Completed), started 5m19s ago; version 4.21.35 is Partial (want Completed), started 6m43s ago; version 4.21.35 is Partial (want Completed), started 8m41s ago; version 4.21.35 is Partial (want Completed), started 12m14s ago; version 4.21.35 is Partial (want Completed), started 13m25s ago; version 4.21.35 is Partial (want Completed), started 13m48s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 16:19	PR #7068	2103519715358937088

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...

full failure pattern:

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional token... message truncated; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xc0006377a0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-8lt5rxb4tbjz" to finish creating: GET https://rp.j7857664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ed95f0cc-a570-4ff6-b5aa-553b0f2e32b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ed95f0cc-a570-4ff6-b5aa-553b0f2e32b6",
      "name": "ed95f0cc-a570-4ff6-b5aa-553b0f2e32b6",
      "status": "Failed",
      "startTime": "2026-09-25T04:05:25.290345016Z",
      "endTime": "2026-09-25T04:24:25.319865016Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-6qhsv-d96qd: Failed: interfaces failed to create or update. err: failed to get existing resource ephemeral-osdisk-8lt5rxb4tbjz--managed/ephemeral-disk-ephemeral-np-6qhsv-d96qd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-8lt5rxb4tbjz--managed/providers/Microsoft.Network/networkInterfaces/ephemeral-disk-ephemeral-np-6qhsv-d96qd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '10' seconds after additional token... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xc0006377a0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-8lt5rxb4tbjz" to finish creating: GET https://rp.j7857664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ed95f0cc-a570-4ff6-b5aa-553b0f2e32b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ed95f0cc-a570-4ff6-b5aa-553b0f2e32b6",
      "name": "ed95f0cc-a570-4ff6-b5aa-553b0f2e32b6",
      "status": "Failed",
      "startTime": "2026-09-25T04:05:25.290345016Z",
      "endTime": "2026-09-25T04:24:25.319865016Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-6qhsv-d96qd: Failed: interfaces failed to create or update. err: failed to get existing resource ephemeral-osdisk-8lt5rxb4tbjz--managed/ephemeral-disk-ephemeral-np-6qhsv-d96qd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-8lt5rxb4tbjz--managed/providers/Microsoft.Network/networkInterfaces/ephemeral-disk-ephemeral-np-6qhsv-d96qd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '10' seconds after additional token... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:58	PR #7135	2103318277487857664

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want >= 2 (autoscaling min); node pool condition AllNodesHealthy is ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want >= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-nfdpj-<id> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-nfdpj-<id> to exist; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc00134acc0>: 
    VerifyNodesReady failed: there are not ready nodes: [agg-cluster-agg-np-a-nfdpj-h6dps]
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-z8btrh6l7hvp" to finish creating: GET https://rp.j2523264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9dfb13f2-af5d-471c-b272-b1c35505253f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9dfb13f2-af5d-471c-b272-b1c35505253f",
      "name": "9dfb13f2-af5d-471c-b272-b1c35505253f",
      "status": "Failed",
      "startTime": "2026-09-25T12:51:15.748310239Z",
      "endTime": "2026-09-25T13:10:18.43513357Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-nfdpj-h6dps: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-z8btrh6l7hvp--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-nfdpj-h6dps to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-nfdpj-h6dps: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-z8btrh6l7hvp--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-nfdpj-h6dps to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc00134acc0>: 
    VerifyNodesReady failed: there are not ready nodes: [agg-cluster-agg-np-a-nfdpj-h6dps]
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-z8btrh6l7hvp" to finish creating: GET https://rp.j2523264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9dfb13f2-af5d-471c-b272-b1c35505253f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9dfb13f2-af5d-471c-b272-b1c35505253f",
      "name": "9dfb13f2-af5d-471c-b272-b1c35505253f",
      "status": "Failed",
      "startTime": "2026-09-25T12:51:15.748310239Z",
      "endTime": "2026-09-25T13:10:18.43513357Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-nfdpj-h6dps: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-z8btrh6l7hvp--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-nfdpj-h6dps to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-nfdpj-h6dps: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-z8btrh6l7hvp--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-nfdpj-h6dps to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 11:30	PR #7165	2103447011792523264

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the ...

full failure pattern:

ERROR CODE: InvalidTemplate; detail message Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'maestroMqttEventHubId'. The parameters defined in the template are: 'databaseName, eventHubId, eventhubEnabled, kustoConsumerGroupName, kustoDataConnectionName, kustoEnabled, kustoName, location'.

Signal:

provision

none

Full failure examples (1)

time=2026-09-25T03:23:26.924Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection description="Step maestroMqttDataConnection\n  Kind: ARM\n  Template: templates/maestro-mqtt-data-connection.bicep\n  Parameters: configurations/maestro-mqtt-data-connection.tmpl.bicepparam"
time=2026-09-25T03:23:27.268Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection
time=2026-09-25T03:23:27.485Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection err="failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/23db75d3c4a6f5fe32de7df35567f808137cc8acac2d401192c396cc43f1121e\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplate\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplate\",\n    \"message\": \"Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'maestroMqttEventHubId'. The parameters defined in the template are: 'databaseName, eventHubId, eventhubEnabled, kustoConsumerGroupName, kustoDataConnectionName, kustoEnabled, kustoName, location'. Please see https://aka.ms/arm-pass-parameter-values for usage details.'.\",\n    \"additionalInfo\": [\n      {\n        \"type\": \"TemplateViolation\",\n        \"info\": {\n          \"lineNumber\": 0,\n          \"linePosition\": 0,\n          \"path\": \"\"\n        }\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/kusto/maestroMqttDataConnection	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:58	PR #7146	2103318212358705152

VerifyImagePulled(acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal) timed out after 3m0.007s: expected image from acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal in namespace acr...

full failure pattern:

VerifyImagePulled(acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal) timed out after 3m0.007s: expected image from acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal in namespace acr-pull-test: image pull errors:

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:334]: failed to pull image from private ACR acrpullcreatebgsbfthf8tqk.azurecr.io — the credential provider did not authenticate the pull
Unexpected error:
    <*fmt.wrapError | 0xc001270ec0>: 
    VerifyImagePulled(acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal) timed out after 3m0.007s: expected image from acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal in namespace acr-pull-test: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 08125a7d-2aeb-4cda-8888-4d2636d60fd2; artifact err: get manifest: build image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 48c612c6-0cab-4944-9d49-fbdaec01dcc2
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:334]: failed to pull image from private ACR acrpullcreatebgsbfthf8tqk.azurecr.io — the credential provider did not authenticate the pull
Unexpected error:
    <*fmt.wrapError | 0xc001270ec0>: 
    VerifyImagePulled(acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal) timed out after 3m0.007s: expected image from acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal in namespace acr-pull-test: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullcreatebgsbfthf8tqk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 08125a7d-2aeb-4cda-8888-4d2636d60fd2; artifact err: get manifest: build image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 48c612c6-0cab-4944-9d49-fbdaec01dcc2
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 18:09	PR #7178	2103547336255868928

VerifyNodesReady failed: there are not ready nodes: [<node>]

full failure pattern:

VerifyNodesReady failed: there are not ready nodes: [<node>]

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:444]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0014697a0>: 
    VerifyNodesReady failed: there are not ready nodes: [agg-cluster-agg-np-a-5bd8h-kb8d6]
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:444]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0014697a0>: 
    VerifyNodesReady failed: there are not ready nodes: [agg-cluster-agg-np-a-5bd8h-kb8d6]
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 16:14	PR #7053	2103518634033811456

failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1

Signal:

provision

none

Full failure examples (1)

time=2026-09-25T13:12:26.954Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 description="Step mitigations\n Kind: Helm\n"
time=2026-09-25T13:12:27.437Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4]"
time=2026-09-25T13:12:27.438Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-09-25T13:12:27.610Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 namespace=aro-hcp-mitigations
time=2026-09-25T13:12:27.765Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 dryRun=false
time=2026-09-25T13:12:27.765Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-09-25T13:12:27.816Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-09-25T13:13:20.164Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-09-25T13:13:20.164Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/mitigations (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 12:24	PR #7133	2103460622090899456

ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid

full failure pattern:

ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid

Signal:

provision

none

Full failure examples (3)

time=2026-09-26T00:07:26.244Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:07:27.744Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:07:28.475Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=c6f6a0e9ed66576db75e4f5edd9325023ba1bd00cc1573f55cc43b013f3d8254 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5401344%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc6f6a0e9ed66576db75e4f5edd9325023ba1bd00cc1573f55cc43b013f3d8254
time=2026-09-26T00:10:59.556Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5401344/providers/Microsoft.Resources/deployments/c6f6a0e9ed66576db75e4f5edd9325023ba1bd00cc1573f55cc43b013f3d8254/operationStatuses/08584112256374751743\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5401344/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5401344/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j5401344\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report c3d253d0-b4f4-4213-be26-e137eaf24c7f:9/26/2026 12:08:41 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T00:12:59.562Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:13:00.295Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:13:01.023Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=2c4f9afc32064e7fbde0098252476ca4f2c555e802c124a268312269ae74ced5 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5401344%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2c4f9afc32064e7fbde0098252476ca4f2c555e802c124a268312269ae74ced5
time=2026-09-26T00:29:35.447Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5401344/providers/Microsoft.Resources/deployments/2c4f9afc32064e7fbde0098252476ca4f2c555e802c124a268312269ae74ced5/operationStatuses/08584112253049438015\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5401344/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5401344/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j5401344\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 6d273900-3e80-430e-9e9b-cc980a819241:9/26/2026 12:28:54 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T00:31:35.447Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:31:46.002Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:31:47.038Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=b9caeeac5f96c466c7656ddeb7452f11f9d047bb47865f52e598d619e4e0310e portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5401344%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb9caeeac5f96c466c7656ddeb7452f11f9d047bb47865f52e598d619e4e0310e
time=2026-09-26T00:37:25.849Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

time=2026-09-26T00:05:44.311Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:05:46.720Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:05:48.319Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f4b607d0d02c46670f1d1fdf0696848d6417d4c2382cde6526a8d79614cac1a7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9084672%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff4b607d0d02c46670f1d1fdf0696848d6417d4c2382cde6526a8d79614cac1a7
time=2026-09-26T00:08:19.373Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9084672/providers/Microsoft.Resources/deployments/f4b607d0d02c46670f1d1fdf0696848d6417d4c2382cde6526a8d79614cac1a7/operationStatuses/08584112257382371931\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9084672/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9084672/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j9084672\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 0fdc339d-1c4e-4de3-b77d-f47ecab3fef7:9/26/2026 12:07:04 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T00:10:19.375Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:10:20.118Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:10:21.694Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=3269581d185d03931c9b79cc66339132c519a978e6f6cda46cb80eebc3d7837d portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9084672%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3269581d185d03931c9b79cc66339132c519a978e6f6cda46cb80eebc3d7837d
time=2026-09-26T00:25:57.330Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9084672/providers/Microsoft.Resources/deployments/3269581d185d03931c9b79cc66339132c519a978e6f6cda46cb80eebc3d7837d/operationStatuses/08584112254648539999\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9084672/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9084672/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j9084672\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 46b4a858-284e-4e1b-b169-e197dc0eb47c:9/26/2026 12:25:39 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T00:27:57.330Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:27:58.460Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:27:59.909Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f623b6f27eafd69dba9c767544d134313b90ae203320d5f11a93ac200693032d portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9084672%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff623b6f27eafd69dba9c767544d134313b90ae203320d5f11a93ac200693032d
time=2026-09-26T00:35:43.999Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

time=2026-09-26T00:11:52.987Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:11:54.672Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:11:55.459Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=8e7952d540cbe777236c9644310d488aefae2fd308040a879252afa860e9ec88 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9889920%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8e7952d540cbe777236c9644310d488aefae2fd308040a879252afa860e9ec88
time=2026-09-26T00:14:56.175Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9889920/providers/Microsoft.Resources/deployments/8e7952d540cbe777236c9644310d488aefae2fd308040a879252afa860e9ec88/operationStatuses/08584112253704864966\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9889920/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9889920/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j9889920\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 9ff84196-1c4e-41dd-a6f2-0a17e70f2462:9/26/2026 12:12:58 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T00:16:56.181Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:16:56.918Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:16:58.061Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=8129d6c1c6edbc3c4d6f196bbe14c57681a45bafc6a3165af424b44cfbdb204a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9889920%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8129d6c1c6edbc3c4d6f196bbe14c57681a45bafc6a3165af424b44cfbdb204a
time=2026-09-26T00:32:31.996Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9889920/providers/Microsoft.Resources/deployments/8129d6c1c6edbc3c4d6f196bbe14c57681a45bafc6a3165af424b44cfbdb204a/operationStatuses/08584112250680737189\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9889920/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9889920/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j9889920\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report c6cc3291-7c3b-48c1-a2de-ec48a0327d16:9/26/2026 12:32:03 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T00:34:31.997Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T00:34:33.135Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T00:34:33.952Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=1e08d522a757546e7297fa1b2c3574a69227d0f73489fac16a6eebd7e354f5a5 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9889920%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F1e08d522a757546e7297fa1b2c3574a69227d0f73489fac16a6eebd7e354f5a5
time=2026-09-26T00:41:52.735Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra	43

Affected runs (43)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:46	PR #7185	2103632247545401344
2026-09-25 23:40	PR #6577	2103630683309084672
2026-09-25 23:38	PR #7187	2103630333239889920
2026-09-25 23:36	PR #7188	2103629622468939776
2026-09-25 23:07	PR #7178	2103622374401249280
2026-09-25 22:49	PR #7141	2103618033778954240
2026-09-25 22:49	PR #7174	2103618006578892800
2026-09-25 22:27	PR #7182	2103612428464951296
2026-09-25 22:27	PR #7183	2103612411738066944
2026-09-25 22:19	PR #7172	2103610246076305408
2026-09-25 22:18	PR #7176	2103610217940914176
2026-09-25 22:18	PR #7157	2103610216275775488
2026-09-25 22:18	PR #7068	2103610212916137984
2026-09-25 22:01	PR #7141	2103605913662263296
2026-09-25 21:50	PR #7185	2103603168511266816
2026-09-25 21:34	PR #7074	2103599075277410304
2026-09-25 21:20	PR #7164	2103595508659392512
2026-09-25 21:20	PR #7165	2103595507854086144
2026-09-25 21:20	PR #7054	2103595503156465664
2026-09-25 21:20	PR #7142	2103595459770585088
2026-09-25 21:20	PR #6821	2103595446143291392
2026-09-25 21:20	PR #7168	2103595427583496192
2026-09-25 21:19	PR #7178	2103595282880008192
2026-09-25 21:17	PR #6577	2103594698844147712
2026-09-25 20:17	PR #7068	2103579624884670464
2026-09-25 20:17	PR #7157	2103579604630376448
2026-09-25 19:59	PR #7181	2103575188418334720
2026-09-25 19:47	PR #7180	2103572043327868928
2026-09-25 19:20	PR #7141	2103565386573156352
2026-09-25 19:17	PR #6821	2103564555174023168
2026-09-25 19:17	PR #7174	2103564545086722048
2026-09-25 19:17	PR #7165	2103564524710793216
2026-09-25 19:17	PR #7054	2103564515059699712
2026-09-25 07:08	PR #7074	2103381081762828288
2026-09-25 07:07	PR #7150	2103380979837046784
2026-09-25 06:22	PR #7123	2103369630306275328
2026-09-25 06:21	PR #7135	2103369316203237376
2026-09-25 04:10	PR #7156	2103336399548715008
2026-09-25 02:58	PR #7053	2103318302158753792
2026-09-25 02:43	PR #7155	2103314299182125056
2026-09-25 02:07	PR #6577	2103305260238901248
2026-09-25 00:34	PR #7149	2103281892986982400
2026-09-25 00:14	PR #6577	2103277014529085440

alert [svc] FrontendPathMedianLatency fired

full failure pattern:

alert [svc] FrontendPathMedianLatency fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:39:56Z
  Ended:       2026-09-25T19:45:57Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7879168-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7879168-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:33:39Z
  Ended:       2026-09-25T19:39:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8124672-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8124672-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T17:55:16Z
  Ended:       2026-09-25T18:03:15Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1821696-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1821696-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire	30

Affected runs (30)

Date (UTC)	Associated PR	Prow job
2026-09-25 18:15	PR #7168	2103549055647879168
2026-09-25 18:09	PR #7164	2103547560068124672
2026-09-25 16:19	PR #7165	2103519763601821696
2026-09-25 16:19	PR #6993	2103519758426050560
2026-09-25 16:19	PR #7054	2103519728415805440
2026-09-25 16:19	PR #7068	2103519715358937088
2026-09-25 16:18	PR #6821	2103519651102199808
2026-09-25 16:14	PR #7053	2103518634033811456
2026-09-25 14:31	PR #7146	2103492665126424576
2026-09-25 14:23	PR #7157	2103490540099080192
2026-09-25 14:12	PR #6577	2103487715302445056
2026-09-25 12:31	PR #7155	2103462425171857408
2026-09-25 12:28	PR #6970	2103461761565855744
2026-09-25 12:24	PR #7123	2103460627115675648
2026-09-25 11:48	PR #6577	2103451563560275968
2026-09-25 11:43	PR #7157	2103450358956167168
2026-09-25 11:30	PR #7165	2103447011792523264
2026-09-25 09:59	PR #7003	2103424207261011968
2026-09-25 09:58	PR #7068	2103423928134275072
2026-09-25 09:58	PR #6993	2103423855413432320
2026-09-25 09:18	PR #7165	2103413943887204352
2026-09-25 09:13	PR #6688	2103412592713142272
2026-09-25 09:07	PR #7071	2103411013251502080
2026-09-25 08:56	PR #7157	2103408234294415360
2026-09-25 08:11	PR #7074	2103397005047894016
2026-09-25 02:58	PR #7150	2103318253316083712
2026-09-25 02:58	PR #6993	2103318216586563584
2026-09-25 02:58	PR #7068	2103318217182154752
2026-09-25 02:58	PR #7123	2103318164392644608
2026-09-25 02:05	PR #7118	2103304771547959296

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T17:47:24Z
  Ended:       2026-09-25T17:56:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6050560-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T16:53:53Z
  Ended:       2026-09-25T17:07:51Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4957824-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T15:34:43Z
  Ended:       2026-09-25T15:41:44Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0090112-svc", component="backend", name="createnodepoolscopedreaddesires", region="canadacentral", severity="warning"
  Description: Backend controller workqueue createnodepoolscopedreaddesires has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	12

Affected runs (12)

Date (UTC)	Associated PR	Prow job
2026-09-25 16:19	PR #6993	2103519758426050560
2026-09-25 15:29	PR #7143	2103507194874957824
2026-09-25 14:19	PR #7074	2103489696960090112
2026-09-25 14:12	PR #6577	2103487715302445056
2026-09-25 14:02	PR #6920	2103485377787465728
2026-09-25 12:24	PR #7123	2103460627115675648
2026-09-25 11:43	PR #7157	2103450358956167168
2026-09-25 09:58	PR #7068	2103423928134275072
2026-09-25 09:39	PR #7009	2103419069259583488
2026-09-25 02:58	PR #7135	2103318277487857664
2026-09-25 02:58	PR #7068	2103318217182154752
2026-09-25 02:06	PR #7141	2103305015505457152

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T15:28:09Z
  Ended:       2026-09-25T15:59:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9080192-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t244r5142spe6g2cg9o8mae3d7hdnhu-t6n2x3b5n5t1m1v", pod="router-5664b4f57c-gtjxn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t244r5142spe6g2cg9o8mae3d7hdnhu-t6n2x3b5n5t1m1v/router-5664b4f57c-gtjxn has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-25T15:28:09Z
  Ended:       2026-09-25T15:58:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9080192-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t244nbkhcsoo0pc5tudo89g01hj3nvr-z3c3n6b0x0c8b7t", pod="router-7588d494ff-mm7tn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t244nbkhcsoo0pc5tudo89g01hj3nvr-z3c3n6b0x0c8b7t/router-7588d494ff-mm7tn has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-25T15:30:08Z
  Ended:       2026-09-25T15:59:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9080192-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t244qbe5vrekus0gjf8imef3oj3a1ab-v7v0u3y9z5d5a0r", pod="router-6ffb8bdb9f-rtb5z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t244qbe5vrekus0gjf8imef3oj3a1ab-v7v0u3y9z5d5a0r/router-6ffb8bdb9f-rtb5z has been in a non-ready state for longer than 5 minutes.

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T13:29:51Z
  Ended:       2026-09-25T13:50:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5855744-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t22eefold2tc0okjf84jebs6os67poh-f3x9e9n2u8k1f3x", pod="router-866f6b9b68-5nwqm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t22eefold2tc0okjf84jebs6os67poh-f3x9e9n2u8k1f3x/router-866f6b9b68-5nwqm has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T13:27:57Z
  Ended:       2026-09-25T13:56:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5675648-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t22c27eesbt60en98i4csdbemrlhr44-v20261001", pod="router-db8fb6c46-6jtmm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t22c27eesbt60en98i4csdbemrlhr44-v20261001/router-db8fb6c46-6jtmm has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-25T13:41:56Z
  Ended:       2026-09-25T13:47:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5675648-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t22cvuckounk62fjihgg1h5pnaiu1h5-y1b7j9n3p2c4e0z", pod="konnectivity-agent-7848799974-nnq7f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t22cvuckounk62fjihgg1h5pnaiu1h5-y1b7j9n3p2c4e0z/konnectivity-agent-7848799974-nnq7f has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-25 14:23	PR #7157	2103490540099080192
2026-09-25 12:28	PR #6970	2103461761565855744
2026-09-25 12:24	PR #7123	2103460627115675648
2026-09-25 11:48	PR #6577	2103451563560275968
2026-09-25 08:08	PR #6821	2103396091608174592
2026-09-25 04:50	PR #7157	2103346399469899776

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc00150e5d0>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-8x29dg6bs8tb" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-8x29dg6bs8tb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc00150e5d0>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-8x29dg6bs8tb" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-8x29dg6bs8tb, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc002c1e3f0>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-rrf2jq8hklbh" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-rrf2jq8hklbh, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc002c1e3f0>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-rrf2jq8hklbh" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-rrf2jq8hklbh, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc000d56060>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-ljjsv8j98zgt" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-ljjsv8j98zgt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc000d56060>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-ljjsv8j98zgt" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-ljjsv8j98zgt, error: context deadline exceeded
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	5
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-25 17:17	PR #7157	2103534322014228480
2026-09-25 14:20	PR #7142	2103489841365782528
2026-09-25 12:24	PR #7123	2103460627115675648
2026-09-25 12:10	PR #7009	2103457038066520064
2026-09-25 08:08	PR #6821	2103396091608174592
2026-09-25 02:06	PR #7141	2103305015505457152

alert [hcp] userJourneyEtcdReadLatencyP991h5m fired

full failure pattern:

alert [hcp] userJourneyEtcdReadLatencyP991h5m fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T21:24:49Z
  Ended:       2026-09-25T21:44:50Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j8545408-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t29brjns56m1264ar7oq3gbmbspvafd-t7i2h8r3z4p7q4p", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j8545408-mgmt-1
Namespace: ocm-arohcpci01-2t29brjns56m1264ar7oq3gbmbspvafd-t7i2h8r3z4p7q4p

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T20:49:53Z
  Ended:       2026-09-25T21:05:53Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j2365184-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2365184-mgmt-1
Namespace: ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001

Firing 2:
  State:       Resolved
  Started:     2026-09-25T21:08:52Z
  Ended:       2026-09-25T21:16:51Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j2365184-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2365184-mgmt-1
Namespace: ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:55:08Z
  Ended:       2026-09-25T20:04:06Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j6147712-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t27vir57rapi3gvc4mcto7vr7g9fer5-cilium-cluster", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j6147712-mgmt-1
Namespace: ocm-arohcpci01-2t27vir57rapi3gvc4mcto7vr7g9fer5-cilium-cluster

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneyEtcdReadLatencyP991h5m does not fire	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-25 20:17	PR #7169	2103579630278545408
2026-09-25 19:56	PR #7172	2103574351302365184
2026-09-25 18:46	PR #7053	2103556819036147712
2026-09-25 16:47	PR #7170	2103526925438816256
2026-09-25 16:19	PR #7165	2103519763601821696
2026-09-25 16:18	PR #6821	2103519651102199808

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:137]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000f7d980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-zlq2sfq4grml" to finish creating: GET https://rp.j8124672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0ffedb5-9117-42d3-a6d0-4533097f5580
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0ffedb5-9117-42d3-a6d0-4533097f5580",
      "name": "f0ffedb5-9117-42d3-a6d0-4533097f5580",
      "status": "Failed",
      "startTime": "2026-09-25T19:12:52.186530809Z",
      "endTime": "2026-09-25T19:31:54.854421453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-bfqvj-mnbtw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-zlq2sfq4grml--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bfqvj-mnbtw to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-bfqvj-mnbtw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-zlq2sfq4grml--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bfqvj-mnbtw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:137]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000f7d980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-zlq2sfq4grml" to finish creating: GET https://rp.j8124672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0ffedb5-9117-42d3-a6d0-4533097f5580
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0ffedb5-9117-42d3-a6d0-4533097f5580",
      "name": "f0ffedb5-9117-42d3-a6d0-4533097f5580",
      "status": "Failed",
      "startTime": "2026-09-25T19:12:52.186530809Z",
      "endTime": "2026-09-25T19:31:54.854421453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-bfqvj-mnbtw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-zlq2sfq4grml--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bfqvj-mnbtw to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-bfqvj-mnbtw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-zlq2sfq4grml--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-bfqvj-mnbtw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fsqv8x" for cluster "cluster-candidate-4-23-fsqv8x"
Unexpected error:
    <*fmt.wrapError | 0xc000ae0220>: 
    failed to create NodePool np-fsqv8x: failed waiting for nodepool="np-fsqv8x" for cluster "cluster-candidate-4-23-fsqv8x" in resourcegroup="rg-candidate-4-23-pzhj9mqnvlqb" to finish creating: GET https://rp.j4957824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18751211-9c50-4f3d-bb4a-0903f4b13e1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18751211-9c50-4f3d-bb4a-0903f4b13e1e",
      "name": "18751211-9c50-4f3d-bb4a-0903f4b13e1e",
      "status": "Failed",
      "startTime": "2026-09-25T16:52:45.296063512Z",
      "endTime": "2026-09-25T17:11:55.439732808Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm to exist\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm to exist\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fsqv8x" for cluster "cluster-candidate-4-23-fsqv8x"
Unexpected error:
    <*fmt.wrapError | 0xc000ae0220>: 
    failed to create NodePool np-fsqv8x: failed waiting for nodepool="np-fsqv8x" for cluster "cluster-candidate-4-23-fsqv8x" in resourcegroup="rg-candidate-4-23-pzhj9mqnvlqb" to finish creating: GET https://rp.j4957824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18751211-9c50-4f3d-bb4a-0903f4b13e1e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18751211-9c50-4f3d-bb4a-0903f4b13e1e",
      "name": "18751211-9c50-4f3d-bb4a-0903f4b13e1e",
      "status": "Failed",
      "startTime": "2026-09-25T16:52:45.296063512Z",
      "endTime": "2026-09-25T17:11:55.439732808Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm to exist\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-4fmpm to exist\nMachine p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-pzhj9mqnvlqb-candidate-fsqv8x--managed/providers/Microsoft.Compute/virtualMachines/p7t0b7i3k0p8m3o-np-fsqv8x-kg66c-mhmlq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000e138c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-l4sqwq4856qf" to finish creating: GET https://rp.j6006272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d70af1e-4af9-4f19-a168-298a07db118f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d70af1e-4af9-4f19-a168-298a07db118f",
      "name": "4d70af1e-4af9-4f19-a168-298a07db118f",
      "status": "Failed",
      "startTime": "2026-09-25T16:36:46.813798448Z",
      "endTime": "2026-09-25T16:55:47.121974819Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine acr-pull-create-np-1-pnchl-ppgqx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-l4sqwq4856qf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-pnchl-ppgqx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine acr-pull-create-np-1-pnchl-ppgqx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-l4sqwq4856qf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-pnchl-ppgqx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000e138c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-l4sqwq4856qf" to finish creating: GET https://rp.j6006272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d70af1e-4af9-4f19-a168-298a07db118f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d70af1e-4af9-4f19-a168-298a07db118f",
      "name": "4d70af1e-4af9-4f19-a168-298a07db118f",
      "status": "Failed",
      "startTime": "2026-09-25T16:36:46.813798448Z",
      "endTime": "2026-09-25T16:55:47.121974819Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine acr-pull-create-np-1-pnchl-ppgqx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-l4sqwq4856qf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-pnchl-ppgqx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine acr-pull-create-np-1-pnchl-ppgqx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-l4sqwq4856qf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-pnchl-ppgqx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-25 18:09	PR #7164	2103547560068124672
2026-09-25 15:29	PR #7143	2103507194874957824
2026-09-25 15:23	PR #7176	2103505655406006272
2026-09-25 14:20	PR #7142	2103489841365782528
2026-09-25 14:02	PR #7165	2103485306291359744
2026-09-25 14:01	PR #7172	2103485064313573376

alert [hcp] userJourneyEtcdWriteLatencyP991h5m fired

full failure pattern:

alert [hcp] userJourneyEtcdWriteLatencyP991h5m fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T20:49:54Z
  Ended:       2026-09-25T21:04:52Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2365184-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2365184-mgmt-1
Namespace: ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001

Firing 2:
  State:       Resolved
  Started:     2026-09-25T21:08:57Z
  Ended:       2026-09-25T21:16:52Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2365184-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2365184-mgmt-1
Namespace: ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:26:52Z
  Ended:       2026-09-25T19:33:50Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j7879168-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t27f39puke0sb3qe61e8d6ncedrr4fe-s7p5n9n6h7p3b7d", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j7879168-mgmt-1
Namespace: ocm-arohcpci01-2t27f39puke0sb3qe61e8d6ncedrr4fe-s7p5n9n6h7p3b7d

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T18:02:18Z
  Ended:       2026-09-25T18:09:19Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j8816256-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t2671elvg1k4fc73r11qe4v243n9j7f-v20260901", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j8816256-mgmt-2
Namespace: ocm-arohcpci01-2t2671elvg1k4fc73r11qe4v243n9j7f-v20260901

Firing 2:
  State:       Resolved
  Started:     2026-09-25T18:18:19Z
  Ended:       2026-09-25T18:24:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j8816256-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t2670enml25if5q1da1ra9v7u3tj5tb-q6q2g5a5a3y2p8o", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j8816256-mgmt-1
Namespace: ocm-arohcpci01-2t2670enml25if5q1da1ra9v7u3tj5tb-q6q2g5a5a3y2p8o

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneyEtcdWriteLatencyP991h5m does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-09-25 19:56	PR #7172	2103574351302365184
2026-09-25 18:15	PR #7168	2103549055647879168
2026-09-25 16:47	PR #7170	2103526925438816256
2026-09-25 16:19	PR #6993	2103519758426050560
2026-09-25 16:18	PR #6821	2103519651102199808

alert [hcp] userJourneyEtcdReadLatencyP996h30m fired

full failure pattern:

alert [hcp] userJourneyEtcdReadLatencyP996h30m fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T21:27:50Z
  Ended:       2026-09-25T22:09:49Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j8545408-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t29brjns56m1264ar7oq3gbmbspvafd-t7i2h8r3z4p7q4p", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j8545408-mgmt-1
Namespace: ocm-arohcpci01-2t29brjns56m1264ar7oq3gbmbspvafd-t7i2h8r3z4p7q4p

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T20:52:51Z
  Ended:       2026-09-25T21:41:55Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j2365184-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2365184-mgmt-1
Namespace: ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:58:05Z
  Ended:       2026-09-25T20:31:09Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j6147712-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t27vir57rapi3gvc4mcto7vr7g9fer5-cilium-cluster", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j6147712-mgmt-1
Namespace: ocm-arohcpci01-2t27vir57rapi3gvc4mcto7vr7g9fer5-cilium-cluster

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneyEtcdReadLatencyP996h30m does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-09-25 20:17	PR #7169	2103579630278545408
2026-09-25 19:56	PR #7172	2103574351302365184
2026-09-25 18:46	PR #7053	2103556819036147712
2026-09-25 16:19	PR #7165	2103519763601821696
2026-09-25 16:18	PR #6821	2103519651102199808

alert [hcp] userJourneyEtcdWriteLatencyP996h30m fired

full failure pattern:

alert [hcp] userJourneyEtcdWriteLatencyP996h30m fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T20:52:54Z
  Ended:       2026-09-25T21:41:59Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j2365184-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2365184-mgmt-1
Namespace: ocm-arohcpci01-2t28rm7oll3gsirl5v5655vkmbilcj89-v20261001

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T18:05:20Z
  Ended:       2026-09-25T18:34:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j8816256-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t2671elvg1k4fc73r11qe4v243n9j7f-v20260901", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j8816256-mgmt-2
Namespace: ocm-arohcpci01-2t2671elvg1k4fc73r11qe4v243n9j7f-v20260901

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T17:39:22Z
  Ended:       2026-09-25T18:20:24Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j6050560-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t260i936l6vnaoo8r9a7o3fqfkl3jcc-f6f8f6c5n1j9d4v", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j6050560-mgmt-1
Namespace: ocm-arohcpci01-2t260i936l6vnaoo8r9a7o3fqfkl3jcc-f6f8f6c5n1j9d4v

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneyEtcdWriteLatencyP996h30m does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-25 19:56	PR #7172	2103574351302365184
2026-09-25 16:47	PR #7170	2103526925438816256
2026-09-25 16:19	PR #6993	2103519758426050560
2026-09-25 16:18	PR #6821	2103519651102199808

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:24:04Z
  Ended:       2026-09-25T19:55:01Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7879168-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7879168/providers/microsoft.monitor/accounts/hcps-j7879168", namespace="ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ftvv6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-25T19:24:04Z
  Ended:       2026-09-25T19:40:02Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7879168-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7879168/providers/microsoft.monitor/accounts/hcps-j7879168", namespace="ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ftvv6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T17:47:38Z
  Ended:       2026-09-25T18:15:38Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3758208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.74:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3758208/providers/microsoft.monitor/accounts/hcps-j3758208", namespace="ocm-arohcpci01-2t262eoagocot786h611d0fvg0ggb7j6-acr-pull-create", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6vl2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t262eoagocot786h611d0fvg0ggb7j6-acr-pull-create/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-25T17:57:37Z
  Ended:       2026-09-25T18:19:42Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3758208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.74:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3758208/providers/microsoft.monitor/accounts/hcps-j3758208", namespace="ocm-arohcpci01-2t262eoagocot786h611d0fvg0ggb7j6-acr-pull-create", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6vl2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t262eoagocot786h611d0fvg0ggb7j6-acr-pull-create/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T14:11:33Z
  Ended:       2026-09-25T14:34:32Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7044096-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.121:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7044096/providers/microsoft.monitor/accounts/hcps-j7044096", namespace="ocm-arohcpci01-2t22r2i2q615199mla8363r7atbm0o71-z4i0x9u5g9q8w6e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-m9x86", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t22r2i2q615199mla8363r7atbm0o71-z4i0x9u5g9q8w6e/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-25 18:15	PR #7168	2103549055647879168
2026-09-25 16:35	PR #7174	2103523744113758208
2026-09-25 13:05	PR #7054	2103470957527044096

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machine...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute

Signal:

e2e

INT, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc00148e780>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-l6zhdc" in resourcegroup="rg-np-version-upgrade-l6zhdc-jh8xxsghsgf8" to finish creating: GET https://rp.j5675648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8325ef8e-dd78-4b1c-8d7e-a8b5d077aa5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8325ef8e-dd78-4b1c-8d7e-a8b5d077aa5a",
      "name": "8325ef8e-dd78-4b1c-8d7e-a8b5d077aa5a",
      "status": "Failed",
      "startTime": "2026-09-25T13:29:34.053488486Z",
      "endTime": "2026-09-25T13:48:39.892100532Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l6zhdc-jh8xxsghsgf8-np-upgrade-l6-78bab067/providers/Microsoft.Compute/virtualMachines/f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l6zhdc-jh8xxsghsgf8-np-upgrade-l6-78bab067/providers/Microsoft.Compute/virtualMachines/f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc00148e780>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-l6zhdc" in resourcegroup="rg-np-version-upgrade-l6zhdc-jh8xxsghsgf8" to finish creating: GET https://rp.j5675648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8325ef8e-dd78-4b1c-8d7e-a8b5d077aa5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8325ef8e-dd78-4b1c-8d7e-a8b5d077aa5a",
      "name": "8325ef8e-dd78-4b1c-8d7e-a8b5d077aa5a",
      "status": "Failed",
      "startTime": "2026-09-25T13:29:34.053488486Z",
      "endTime": "2026-09-25T13:48:39.892100532Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l6zhdc-jh8xxsghsgf8-np-upgrade-l6-78bab067/providers/Microsoft.Compute/virtualMachines/f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l6zhdc-jh8xxsghsgf8-np-upgrade-l6-78bab067/providers/Microsoft.Compute/virtualMachines/f0d9h9y2l6m6d5u-npupgrade-4-21-f2rqb-8jjcb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ca7e40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rjcjqxxsgq7f" to finish creating: GET https://rp.j2523264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17378f74-366a-451c-a16d-8f6eab8588d2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17378f74-366a-451c-a16d-8f6eab8588d2",
      "name": "17378f74-366a-451c-a16d-8f6eab8588d2",
      "status": "Failed",
      "startTime": "2026-09-25T12:45:45.224128185Z",
      "endTime": "2026-09-25T13:04:46.064091322Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-rjcjqxxsgq7f--managed/providers/Microsoft.Compute/virtualMachines/u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-rjcjqxxsgq7f--managed/providers/Microsoft.Compute/virtualMachines/u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ca7e40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rjcjqxxsgq7f" to finish creating: GET https://rp.j2523264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17378f74-366a-451c-a16d-8f6eab8588d2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17378f74-366a-451c-a16d-8f6eab8588d2",
      "name": "17378f74-366a-451c-a16d-8f6eab8588d2",
      "status": "Failed",
      "startTime": "2026-09-25T12:45:45.224128185Z",
      "endTime": "2026-09-25T13:04:46.064091322Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-rjcjqxxsgq7f--managed/providers/Microsoft.Compute/virtualMachines/u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-rjcjqxxsgq7f--managed/providers/Microsoft.Compute/virtualMachines/u6o3a4n8w6e1h9n-np-1-n26gj-tdhxm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pqhwsk" for cluster "cluster-candidate-4-20-pqhwsk"
Unexpected error:
    <*fmt.wrapError | 0xc001061ba0>: 
    failed to create NodePool np-pqhwsk: failed waiting for nodepool="np-pqhwsk" for cluster "cluster-candidate-4-20-pqhwsk" in resourcegroup="rg-candidate-4-20-4hfcxgvg226p" to finish creating: GET https://rp.j2644608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ca27daa-447f-4425-a737-272daf2be568
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ca27daa-447f-4425-a737-272daf2be568",
      "name": "2ca27daa-447f-4425-a737-272daf2be568",
      "status": "Failed",
      "startTime": "2026-09-25T04:05:18.601636409Z",
      "endTime": "2026-09-25T04:24:23.185191796Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-4hfcxgvg226p-candidate-pqhwsk--managed/providers/Microsoft.Compute/virtualMachines/j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-4hfcxgvg226p-candidate-pqhwsk--managed/providers/Microsoft.Compute/virtualMachines/j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pqhwsk" for cluster "cluster-candidate-4-20-pqhwsk"
Unexpected error:
    <*fmt.wrapError | 0xc001061ba0>: 
    failed to create NodePool np-pqhwsk: failed waiting for nodepool="np-pqhwsk" for cluster "cluster-candidate-4-20-pqhwsk" in resourcegroup="rg-candidate-4-20-4hfcxgvg226p" to finish creating: GET https://rp.j2644608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ca27daa-447f-4425-a737-272daf2be568
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ca27daa-447f-4425-a737-272daf2be568",
      "name": "2ca27daa-447f-4425-a737-272daf2be568",
      "status": "Failed",
      "startTime": "2026-09-25T04:05:18.601636409Z",
      "endTime": "2026-09-25T04:24:23.185191796Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-4hfcxgvg226p-candidate-pqhwsk--managed/providers/Microsoft.Compute/virtualMachines/j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-4hfcxgvg226p-candidate-pqhwsk--managed/providers/Microsoft.Compute/virtualMachines/j5f0t5a1f6v6f0d-np-pqhwsk-wxvcc-pbr8x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-25 12:24	PR #7123	2103460627115675648
2026-09-25 11:30	PR #7165	2103447011792523264
2026-09-25 02:58	PR #7123	2103318164392644608

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000adbee0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qqrdr5ldkfxf" to finish creating: GET https://rp.j2365184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "name": "e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "status": "Failed",
      "startTime": "2026-09-25T20:55:06.599822055Z",
      "endTime": "2026-09-25T21:14:14.361175634Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000adbee0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qqrdr5ldkfxf" to finish creating: GET https://rp.j2365184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "name": "e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "status": "Failed",
      "startTime": "2026-09-25T20:55:06.599822055Z",
      "endTime": "2026-09-25T21:14:14.361175634Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003af2080>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dnr4nbhwmf28" to finish creating: GET https://rp.j6050560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/845fcbd5-9444-4826-9970-7a1c7611c2b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/845fcbd5-9444-4826-9970-7a1c7611c2b7",
      "name": "845fcbd5-9444-4826-9970-7a1c7611c2b7",
      "status": "Failed",
      "startTime": "2026-09-25T17:37:30.398464058Z",
      "endTime": "2026-09-25T17:56:37.126932559Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9 to exist\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9 to exist\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003af2080>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dnr4nbhwmf28" to finish creating: GET https://rp.j6050560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/845fcbd5-9444-4826-9970-7a1c7611c2b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/845fcbd5-9444-4826-9970-7a1c7611c2b7",
      "name": "845fcbd5-9444-4826-9970-7a1c7611c2b7",
      "status": "Failed",
      "startTime": "2026-09-25T17:37:30.398464058Z",
      "endTime": "2026-09-25T17:56:37.126932559Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9 to exist\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-j7hs9 to exist\nMachine e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dnr4nbhwmf28--managed/providers/Microsoft.Compute/virtualMachines/e4o0a9k9h1t4n3s-np-deletable-7xqlr-shwl7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007fbbe0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-9ngsk5k5bt5f" to finish creating: GET https://rp.j1011968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c50dfee1-939c-4199-9413-eca90594ea6c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c50dfee1-939c-4199-9413-eca90594ea6c",
      "name": "c50dfee1-939c-4199-9413-eca90594ea6c",
      "status": "Failed",
      "startTime": "2026-09-25T11:12:13.33985788Z",
      "endTime": "2026-09-25T11:31:22.686534566Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-9ngsk5k5bt5f--managed/providers/Microsoft.Compute/virtualMachines/z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-9ngsk5k5bt5f--managed/providers/Microsoft.Compute/virtualMachines/z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007fbbe0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-9ngsk5k5bt5f" to finish creating: GET https://rp.j1011968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c50dfee1-939c-4199-9413-eca90594ea6c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c50dfee1-939c-4199-9413-eca90594ea6c",
      "name": "c50dfee1-939c-4199-9413-eca90594ea6c",
      "status": "Failed",
      "startTime": "2026-09-25T11:12:13.33985788Z",
      "endTime": "2026-09-25T11:31:22.686534566Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-9ngsk5k5bt5f--managed/providers/Microsoft.Compute/virtualMachines/z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-9ngsk5k5bt5f--managed/providers/Microsoft.Compute/virtualMachines/z3q2f0g2z5d6n0b-np-delete-main-72qcj-6f66h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-25 19:56	PR #7172	2103574351302365184
2026-09-25 16:19	PR #6993	2103519758426050560
2026-09-25 09:59	PR #7003	2103424207261011968

ERROR CODE: DeploymentFailed; detail code EventHubValidationErrorFound; detail message [BadRequest] Validation Errors found: Table does not exist

full failure pattern:

ERROR CODE: DeploymentFailed; detail code EventHubValidationErrorFound; detail message [BadRequest] Validation Errors found: Table does not exist

Signal:

provision

none

Full failure examples (3)

time=2026-09-25T07:28:19.460Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection description="Step maestroMqttDataConnection\n  Kind: ARM\n  Template: templates/maestro-mqtt-data-connection.bicep\n  Parameters: configurations/maestro-mqtt-data-connection.tmpl.bicepparam"
time=2026-09-25T07:28:19.869Z level=DEBUG+2 msg="Waiting for existing deployment to complete" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection deployment=67437eb973c49795934fc79733f3a01017eeb2994048cb050f5f7bf549795e78
time=2026-09-25T07:28:19.869Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection err="failed to run ARM step: failed to poll previously-existing deployment: pre-existing deployment failed: {\"code\":\"DeploymentFailed\",\"details\":[{\"code\":\"EventHubValidationErrorFound\",\"message\":\"[BadRequest] Validation Errors found: Table does not exist\"}],\"message\":\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\"}"

time=2026-09-25T04:41:01.572Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection description="Step maestroMqttDataConnection\n  Kind: ARM\n  Template: templates/maestro-mqtt-data-connection.bicep\n  Parameters: configurations/maestro-mqtt-data-connection.tmpl.bicepparam"
time=2026-09-25T04:41:01.944Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection
time=2026-09-25T04:41:02.776Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection deployment=be82b2c4bc5e91c9fb05bb7a64319b66570f7844a6d32af17e112fe3dd39a4e7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-kusto-us%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbe82b2c4bc5e91c9fb05bb7a64319b66570f7844a6d32af17e112fe3dd39a4e7
time=2026-09-25T04:41:32.894Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/be82b2c4bc5e91c9fb05bb7a64319b66570f7844a6d32af17e112fe3dd39a4e7/operationStatuses/08584112956230068016\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"EventHubValidationErrorFound\\\",\\r\\n        \\\"message\\\": \\\"[BadRequest] Validation Errors found: Table does not exist\\\"\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-25T02:39:19.860Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection description="Step maestroMqttDataConnection\n  Kind: ARM\n  Template: templates/maestro-mqtt-data-connection.bicep\n  Parameters: configurations/maestro-mqtt-data-connection.tmpl.bicepparam"
time=2026-09-25T02:39:20.257Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection
time=2026-09-25T02:39:20.970Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection deployment=67437eb973c49795934fc79733f3a01017eeb2994048cb050f5f7bf549795e78 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-kusto-us%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F67437eb973c49795934fc79733f3a01017eeb2994048cb050f5f7bf549795e78
time=2026-09-25T02:39:51.103Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=maestroMqttDataConnection err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/67437eb973c49795934fc79733f3a01017eeb2994048cb050f5f7bf549795e78/operationStatuses/08584113029248069745\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"EventHubValidationErrorFound\\\",\\r\\n        \\\"message\\\": \\\"[BadRequest] Validation Errors found: Table does not exist\\\"\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/kusto/maestroMqttDataConnection	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-25 07:08	PR #7146	2103381037483560960
2026-09-25 04:30	PR #7153	2103341305806458880
2026-09-25 02:27	PR #7153	2103310372827041792

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000990ae0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wf95j7vx69b5" to finish creating: GET https://rp.j5675648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2",
      "name": "00381033-aeb7-400a-bd07-e1bd8611cea2",
      "status": "Failed",
      "startTime": "2026-09-25T13:15:51.082389682Z",
      "endTime": "2026-09-25T13:45:00.023728639Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000990ae0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wf95j7vx69b5" to finish creating: GET https://rp.j5675648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2",
      "name": "00381033-aeb7-400a-bd07-e1bd8611cea2",
      "status": "Failed",
      "startTime": "2026-09-25T13:15:51.082389682Z",
      "endTime": "2026-09-25T13:45:00.023728639Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6ghnhlc94h8f/cluster-hs-pre-5wm67n should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010f25a0>: 
    failed to create HCP cluster cluster-hs-pre-5wm67n: failed waiting for cluster="cluster-hs-pre-5wm67n" in resourcegroup="rg-hs-pre--6ghnhlc94h8f" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911",
      "name": "46dab79f-2360-4680-914d-589d803cc911",
      "status": "Failed",
      "startTime": "2026-09-25T05:47:00.77129988Z",
      "endTime": "2026-09-25T06:16:03.97431147Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6ghnhlc94h8f/cluster-hs-pre-5wm67n should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010f25a0>: 
    failed to create HCP cluster cluster-hs-pre-5wm67n: failed waiting for cluster="cluster-hs-pre-5wm67n" in resourcegroup="rg-hs-pre--6ghnhlc94h8f" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911",
      "name": "46dab79f-2360-4680-914d-589d803cc911",
      "status": "Failed",
      "startTime": "2026-09-25T05:47:00.77129988Z",
      "endTime": "2026-09-25T06:16:03.97431147Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c1aa00>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pgng9vz6j8w8" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327",
      "name": "a614d325-8937-4e4c-a91f-47b2564f0327",
      "status": "Failed",
      "startTime": "2026-09-25T05:48:11.218096649Z",
      "endTime": "2026-09-25T06:17:14.62086737Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m30s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c1aa00>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pgng9vz6j8w8" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327",
      "name": "a614d325-8937-4e4c-a91f-47b2564f0327",
      "status": "Failed",
      "startTime": "2026-09-25T05:48:11.218096649Z",
      "endTime": "2026-09-25T06:17:14.62086737Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m30s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 12:24	PR #7123	2103460627115675648
2026-09-25 04:50	PR #7157	2103346399469899776

alert [svc] ServiceCPUDrift fired

full failure pattern:

alert [svc] ServiceCPUDrift fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T03:34:52Z
  Ended:       2026-09-25T03:43:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7908608-mgmt-1", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-78567cf5bf-qmbvj", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-78567cf5bf-qmbvj (namespace kube-applier) on cluster ci01-j7908608-mgmt-1 has averaged 138.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T03:26:19Z
  Ended:       2026-09-25T03:40:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7959296-mgmt-1", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-7dc989b6d9-rg6jc", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-7dc989b6d9-rg6jc (namespace kube-applier) on cluster ci01-j7959296-mgmt-1 has averaged 138.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ServiceCPUDrift does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:07	PR #7148	2103305281587908608
2026-09-25 02:05	PR #7118	2103304771547959296

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T18:02:42Z
  Ended:       2026-09-25T18:15:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3758208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.74:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3758208/providers/microsoft.monitor/accounts/hcps-j3758208", namespace="ocm-arohcpci01-2t262eoagocot786h611d0fvg0ggb7j6-acr-pull-create", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6vl2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t262eoagocot786h611d0fvg0ggb7j6-acr-pull-create/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T15:53:12Z
  Ended:       2026-09-25T15:59:09Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9080192-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9080192/providers/microsoft.monitor/accounts/hcps-j9080192", namespace="ocm-arohcpci01-2t244r5142spe6g2cg9o8mae3d7hdnhu-t6n2x3b5n5t1m1v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-l2hch", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t244r5142spe6g2cg9o8mae3d7hdnhu-t6n2x3b5n5t1m1v/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-25T15:53:12Z
  Ended:       2026-09-25T15:59:09Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9080192-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9080192/providers/microsoft.monitor/accounts/hcps-j9080192", namespace="ocm-arohcpci01-2t244r5142spe6g2cg9o8mae3d7hdnhu-t6n2x3b5n5t1m1v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-l2hch", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t244r5142spe6g2cg9o8mae3d7hdnhu-t6n2x3b5n5t1m1v/router has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 16:35	PR #7174	2103523744113758208
2026-09-25 14:23	PR #7157	2103490540099080192

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc002da48c0>: 
    DNS for route host agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc002da48c0>: 
    DNS for route host agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc002d20140>: 
    DNS for route host agnhost-e2e-sample-app-rfv4s.apps.aro.v20260901.pw4y.j4790144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rfv4s.apps.aro.v20260901.pw4y.j4790144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rfv4s.apps.aro.v20260901.pw4y.j4790144.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc002d20140>: 
    DNS for route host agnhost-e2e-sample-app-rfv4s.apps.aro.v20260901.pw4y.j4790144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-rfv4s.apps.aro.v20260901.pw4y.j4790144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rfv4s.apps.aro.v20260901.pw4y.j4790144.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 20:36	PR #7104	2103584488377290752
2026-09-25 18:42	PR #7142	2103555692374790144

ERROR CODE: 429; provider Microsoft.ManagedIdentity

full failure pattern:

ERROR CODE: 429; provider Microsoft.ManagedIdentity

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000c39160>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-01-21/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000c39160>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-01-21/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000a4b500>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-26/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000a4b500>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-26/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 16:19	PR #7068	2103519715358937088
2026-09-25 08:08	PR #6821	2103396091608174592

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000d12c80>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-tvqthjct49m6" to finish creating: GET https://rp.j7879168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d574528-4ed3-4914-89a8-24518c1433b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d574528-4ed3-4914-89a8-24518c1433b8",
      "name": "5d574528-4ed3-4914-89a8-24518c1433b8",
      "status": "Failed",
      "startTime": "2026-09-25T19:04:01.460224052Z",
      "endTime": "2026-09-25T19:33:07.221268822Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000d12c80>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-tvqthjct49m6" to finish creating: GET https://rp.j7879168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d574528-4ed3-4914-89a8-24518c1433b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d574528-4ed3-4914-89a8-24518c1433b8",
      "name": "5d574528-4ed3-4914-89a8-24518c1433b8",
      "status": "Failed",
      "startTime": "2026-09-25T19:04:01.460224052Z",
      "endTime": "2026-09-25T19:33:07.221268822Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc000d874a0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-vdsj46l8n4nc" to finish creating: GET https://rp.j3758208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc17ba24-1832-41fc-ac72-c4255c121a55
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc17ba24-1832-41fc-ac72-c4255c121a55",
      "name": "bc17ba24-1832-41fc-ac72-c4255c121a55",
      "status": "Failed",
      "startTime": "2026-09-25T17:28:26.003020977Z",
      "endTime": "2026-09-25T17:57:27.213791219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc000d874a0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-vdsj46l8n4nc" to finish creating: GET https://rp.j3758208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc17ba24-1832-41fc-ac72-c4255c121a55
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc17ba24-1832-41fc-ac72-c4255c121a55",
      "name": "bc17ba24-1832-41fc-ac72-c4255c121a55",
      "status": "Failed",
      "startTime": "2026-09-25T17:28:26.003020977Z",
      "endTime": "2026-09-25T17:57:27.213791219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 18:15	PR #7168	2103549055647879168
2026-09-25 16:35	PR #7174	2103523744113758208

ERROR CODE: DeadlineExceeded; provider Microsoft.Network

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000f1a8c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-r2tsxmn2gn6l" to finish creating: GET https://rp.j2445056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d1d1b0c-beb5-491f-977f-de85fe072646
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d1d1b0c-beb5-491f-977f-de85fe072646",
      "name": "1d1d1b0c-beb5-491f-977f-de85fe072646",
      "status": "Failed",
      "startTime": "2026-09-25T15:02:51.1208565Z",
      "endTime": "2026-09-25T15:21:56.904399075Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-x7jf2-994zl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-x7jf2-4qp6m: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-r2tsxmn2gn6l--managed/v20260901-np-1-x7jf2-4qp6m-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-r2tsxmn2gn6l--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-x7jf2-4qp6m-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000f1a8c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-r2tsxmn2gn6l" to finish creating: GET https://rp.j2445056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d1d1b0c-beb5-491f-977f-de85fe072646
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d1d1b0c-beb5-491f-977f-de85fe072646",
      "name": "1d1d1b0c-beb5-491f-977f-de85fe072646",
      "status": "Failed",
      "startTime": "2026-09-25T15:02:51.1208565Z",
      "endTime": "2026-09-25T15:21:56.904399075Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-x7jf2-994zl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-x7jf2-4qp6m: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-r2tsxmn2gn6l--managed/v20260901-np-1-x7jf2-4qp6m-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-r2tsxmn2gn6l--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-x7jf2-4qp6m-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mtg998" for cluster "cluster-candidate-4-21-mtg998"
Unexpected error:
    <*fmt.wrapError | 0xc000d0bcc0>: 
    failed to create NodePool np-mtg998: failed waiting for nodepool="np-mtg998" for cluster "cluster-candidate-4-21-mtg998" in resourcegroup="rg-candidate-4-21-6rjxr7bt59b4" to finish creating: GET https://rp.j7465728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51518391-ba3d-4fba-ad02-341a42607b48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51518391-ba3d-4fba-ad02-341a42607b48",
      "name": "51518391-ba3d-4fba-ad02-341a42607b48",
      "status": "Failed",
      "startTime": "2026-09-25T14:59:49.694049709Z",
      "endTime": "2026-09-25T15:18:56.27468361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c9n6t8n1t1l0b2l-np-mtg998-dk6h7-bfvhp: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-21-6rjxr7bt59b4-candidate-mtg998--managed/c9n6t8n1t1l0b2l-np-mtg998-dk6h7-bfvhp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-6rjxr7bt59b4-candidate-mtg998--managed/providers/Microsoft.Network/networkInterfaces/c9n6t8n1t1l0b2l-np-mtg998-dk6h7-bfvhp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '3' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mtg998" for cluster "cluster-candidate-4-21-mtg998"
Unexpected error:
    <*fmt.wrapError | 0xc000d0bcc0>: 
    failed to create NodePool np-mtg998: failed waiting for nodepool="np-mtg998" for cluster "cluster-candidate-4-21-mtg998" in resourcegroup="rg-candidate-4-21-6rjxr7bt59b4" to finish creating: GET https://rp.j7465728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51518391-ba3d-4fba-ad02-341a42607b48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51518391-ba3d-4fba-ad02-341a42607b48",
      "name": "51518391-ba3d-4fba-ad02-341a42607b48",
      "status": "Failed",
      "startTime": "2026-09-25T14:59:49.694049709Z",
      "endTime": "2026-09-25T15:18:56.27468361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c9n6t8n1t1l0b2l-np-mtg998-dk6h7-bfvhp: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-21-6rjxr7bt59b4-candidate-mtg998--managed/c9n6t8n1t1l0b2l-np-mtg998-dk6h7-bfvhp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-6rjxr7bt59b4-candidate-mtg998--managed/providers/Microsoft.Network/networkInterfaces/c9n6t8n1t1l0b2l-np-mtg998-dk6h7-bfvhp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '3' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 14:12	PR #6577	2103487715302445056
2026-09-25 14:02	PR #6920	2103485377787465728

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-lbsddf with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000a8c360>: 
    failed to create HCP cluster np-version-upgrade-cluster-lbsddf: failed waiting for cluster="np-version-upgrade-cluster-lbsddf" in resourcegroup="rg-np-version-upgrade-lbsddf-tvrng8dc2496" to finish creating: GET https://rp.j7908608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dfd3731e-5e9b-458d-928f-6eea5509a636
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dfd3731e-5e9b-458d-928f-6eea5509a636",
      "name": "dfd3731e-5e9b-458d-928f-6eea5509a636",
      "status": "Failed",
      "startTime": "2026-09-25T02:59:11.878359495Z",
      "endTime": "2026-09-25T03:28:13.63075435Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-lbsddf with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000a8c360>: 
    failed to create HCP cluster np-version-upgrade-cluster-lbsddf: failed waiting for cluster="np-version-upgrade-cluster-lbsddf" in resourcegroup="rg-np-version-upgrade-lbsddf-tvrng8dc2496" to finish creating: GET https://rp.j7908608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dfd3731e-5e9b-458d-928f-6eea5509a636
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dfd3731e-5e9b-458d-928f-6eea5509a636",
      "name": "dfd3731e-5e9b-458d-928f-6eea5509a636",
      "status": "Failed",
      "startTime": "2026-09-25T02:59:11.878359495Z",
      "endTime": "2026-09-25T03:28:13.63075435Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cccea0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-w9hkzhnrs46x" to finish creating: GET https://rp.j7908608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e53f084-0836-43b7-813a-0f6f679c2c37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e53f084-0836-43b7-813a-0f6f679c2c37",
      "name": "7e53f084-0836-43b7-813a-0f6f679c2c37",
      "status": "Failed",
      "startTime": "2026-09-25T02:59:01.927806669Z",
      "endTime": "2026-09-25T03:28:03.710517796Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cccea0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-w9hkzhnrs46x" to finish creating: GET https://rp.j7908608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e53f084-0836-43b7-813a-0f6f679c2c37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e53f084-0836-43b7-813a-0f6f679c2c37",
      "name": "7e53f084-0836-43b7-813a-0f6f679c2c37",
      "status": "Failed",
      "startTime": "2026-09-25T02:59:01.927806669Z",
      "endTime": "2026-09-25T03:28:03.710517796Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0006df780>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-l2fmpdwgxxr7" to finish creating: GET https://rp.j7908608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2db7371-9a1e-4204-9ef5-6f3d7af49462
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2db7371-9a1e-4204-9ef5-6f3d7af49462",
      "name": "c2db7371-9a1e-4204-9ef5-6f3d7af49462",
      "status": "Failed",
      "startTime": "2026-09-25T02:58:44.421041016Z",
      "endTime": "2026-09-25T03:27:53.514379412Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0006df780>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-l2fmpdwgxxr7" to finish creating: GET https://rp.j7908608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2db7371-9a1e-4204-9ef5-6f3d7af49462
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2db7371-9a1e-4204-9ef5-6f3d7af49462",
      "name": "c2db7371-9a1e-4204-9ef5-6f3d7af49462",
      "status": "Failed",
      "startTime": "2026-09-25T02:58:44.421041016Z",
      "endTime": "2026-09-25T03:27:53.514379412Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (24)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:07	PR #7148	2103305281587908608

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-8vfwxckt8249/cluster-candidate-4-23-pksrt4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fc4780>: 
    failed to create HCP cluster cluster-candidate-4-23-pksrt4: failed waiting for cluster="cluster-candidate-4-23-pksrt4" in resourcegroup="rg-candidate-4-23-8vfwxckt8249" to finish creating: GET https://rp.j9080192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d480bcb-375c-4b16-8bda-6aad2a19dfa7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d480bcb-375c-4b16-8bda-6aad2a19dfa7",
      "name": "2d480bcb-375c-4b16-8bda-6aad2a19dfa7",
      "status": "Failed",
      "startTime": "2026-09-25T15:16:44.436730712Z",
      "endTime": "2026-09-25T15:45:52.413815981Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.0 is Partial (want Completed), started 23m23s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-8vfwxckt8249/cluster-candidate-4-23-pksrt4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fc4780>: 
    failed to create HCP cluster cluster-candidate-4-23-pksrt4: failed waiting for cluster="cluster-candidate-4-23-pksrt4" in resourcegroup="rg-candidate-4-23-8vfwxckt8249" to finish creating: GET https://rp.j9080192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d480bcb-375c-4b16-8bda-6aad2a19dfa7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2d480bcb-375c-4b16-8bda-6aad2a19dfa7",
      "name": "2d480bcb-375c-4b16-8bda-6aad2a19dfa7",
      "status": "Failed",
      "startTime": "2026-09-25T15:16:44.436730712Z",
      "endTime": "2026-09-25T15:45:52.413815981Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.0 is Partial (want Completed), started 23m23s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009b48a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ms6xhv5kcfxf" to finish creating: GET https://rp.j9080192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ddffd0dd-0556-4e1c-9602-37c54fdf6ce2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ddffd0dd-0556-4e1c-9602-37c54fdf6ce2",
      "name": "ddffd0dd-0556-4e1c-9602-37c54fdf6ce2",
      "status": "Failed",
      "startTime": "2026-09-25T15:16:59.889674248Z",
      "endTime": "2026-09-25T15:46:02.475687505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009b48a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ms6xhv5kcfxf" to finish creating: GET https://rp.j9080192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ddffd0dd-0556-4e1c-9602-37c54fdf6ce2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ddffd0dd-0556-4e1c-9602-37c54fdf6ce2",
      "name": "ddffd0dd-0556-4e1c-9602-37c54fdf6ce2",
      "status": "Failed",
      "startTime": "2026-09-25T15:16:59.889674248Z",
      "endTime": "2026-09-25T15:46:02.475687505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-tql9hwr48v4l/cluster-candidate-4-20-bwhjc4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001a9c160>: 
    failed to create HCP cluster cluster-candidate-4-20-bwhjc4: failed waiting for cluster="cluster-candidate-4-20-bwhjc4" in resourcegroup="rg-candidate-4-20-tql9hwr48v4l" to finish creating: GET https://rp.j9080192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3415f09-2ec9-41e2-b5a8-2ceeb8dd3ddb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3415f09-2ec9-41e2-b5a8-2ceeb8dd3ddb",
      "name": "d3415f09-2ec9-41e2-b5a8-2ceeb8dd3ddb",
      "status": "Failed",
      "startTime": "2026-09-25T15:16:54.769562709Z",
      "endTime": "2026-09-25T15:46:02.395056967Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.39 is Partial (want Completed), started 23m51s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-tql9hwr48v4l/cluster-candidate-4-20-bwhjc4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001a9c160>: 
    failed to create HCP cluster cluster-candidate-4-20-bwhjc4: failed waiting for cluster="cluster-candidate-4-20-bwhjc4" in resourcegroup="rg-candidate-4-20-tql9hwr48v4l" to finish creating: GET https://rp.j9080192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3415f09-2ec9-41e2-b5a8-2ceeb8dd3ddb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3415f09-2ec9-41e2-b5a8-2ceeb8dd3ddb",
      "name": "d3415f09-2ec9-41e2-b5a8-2ceeb8dd3ddb",
      "status": "Failed",
      "startTime": "2026-09-25T15:16:54.769562709Z",
      "endTime": "2026-09-25T15:46:02.395056967Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.39 is Partial (want Completed), started 23m51s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 14:23	PR #7157	2103490540099080192

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e94378>: 
    resource group "disabled-image-registry-tvns76rrbqk5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-tvns76rrbqk5" to finish deleting: GET https://rp.j6006272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73511aff-bb96-4107-9e34-0950b93079eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73511aff-bb96-4107-9e34-0950b93079eb",
      "name": "73511aff-bb96-4107-9e34-0950b93079eb",
      "status": "Failed",
      "startTime": "2026-09-25T16:34:38.978866583Z",
      "endTime": "2026-09-25T16:58:47.471362642Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t254sr73df6qs4ckqqphfpe2lj40f4b still exists (deletion dispatched at 2026-09-25T16:34:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e94378>: 
    resource group "disabled-image-registry-tvns76rrbqk5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-tvns76rrbqk5" to finish deleting: GET https://rp.j6006272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73511aff-bb96-4107-9e34-0950b93079eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73511aff-bb96-4107-9e34-0950b93079eb",
      "name": "73511aff-bb96-4107-9e34-0950b93079eb",
      "status": "Failed",
      "startTime": "2026-09-25T16:34:38.978866583Z",
      "endTime": "2026-09-25T16:58:47.471362642Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t254sr73df6qs4ckqqphfpe2lj40f4b still exists (deletion dispatched at 2026-09-25T16:34:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 15:23	PR #7176	2103505655406006272

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

INT

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--nrk2cdj4tcn5/cluster-hs-pre-pcqks7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009409a0>: 
    failed to create HCP cluster cluster-hs-pre-pcqks7: failed waiting for cluster="cluster-hs-pre-pcqks7" in resourcegroup="rg-hs-pre--nrk2cdj4tcn5" to finish creating: GET https://rp.j7044096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea",
      "name": "e89da08a-794d-4885-a19f-3eb28678e2ea",
      "status": "Failed",
      "startTime": "2026-09-25T13:47:52.911662059Z",
      "endTime": "2026-09-25T14:16:59.018766333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--nrk2cdj4tcn5/cluster-hs-pre-pcqks7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009409a0>: 
    failed to create HCP cluster cluster-hs-pre-pcqks7: failed waiting for cluster="cluster-hs-pre-pcqks7" in resourcegroup="rg-hs-pre--nrk2cdj4tcn5" to finish creating: GET https://rp.j7044096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea",
      "name": "e89da08a-794d-4885-a19f-3eb28678e2ea",
      "status": "Failed",
      "startTime": "2026-09-25T13:47:52.911662059Z",
      "endTime": "2026-09-25T14:16:59.018766333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 13:05	PR #7054	2103470957527044096

ERROR CODE: DeploymentFailed; detail code InternalServerError; detail message An error occurred.

full failure pattern:

ERROR CODE: DeploymentFailed; detail code InternalServerError; detail message An error occurred.

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:80]: failed to create cluster customer resources for cordon test
Unexpected error:
    <*fmt.wrapError | 0xc000b98ae0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cordon-hcp-cluster-rck8ht" in resourcegroup="rg-cordon-test-tlds99sfrm5j" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cordon-test-tlds99sfrm5j/providers/Microsoft.Resources/deployments/customer-infra-cordon-hcp-cluster-rck8ht/operationStatuses/08584113017077419903
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"InternalServerError\",\r\n        \"message\": \"An error occurred.\",\r\n        \"details\": []\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:80]: failed to create cluster customer resources for cordon test
Unexpected error:
    <*fmt.wrapError | 0xc000b98ae0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cordon-hcp-cluster-rck8ht" in resourcegroup="rg-cordon-test-tlds99sfrm5j" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cordon-test-tlds99sfrm5j/providers/Microsoft.Resources/deployments/customer-infra-cordon-hcp-cluster-rck8ht/operationStatuses/08584113017077419903
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"InternalServerError\",\r\n        \"message\": \"An error occurred.\",\r\n        \"details\": []\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:06	PR #7141	2103305015505457152

alert [svc] FrontendPathLatency fired

full failure pattern:

alert [svc] FrontendPathLatency fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T10:21:49Z
  Ended:       2026-09-25T10:27:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j6461184-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j6461184-svc).
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-25T10:32:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j6461184-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationstatuses/{operationid}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationstatuses/{operationid} exceeded 1 second over the past 30 minutes (cluster ci01-j6461184-svc).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 08:41	PR #7105	2103404569546461184

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:26:00Z
  Ended:       2026-09-25T19:56:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7879168-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7879168/providers/microsoft.monitor/accounts/hcps-j7879168", namespace="ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fd1ced12-4634-450b-be94-fe821eb628d0"
  Description: Pod ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-25T19:26:00Z
  Ended:       2026-09-25T19:55:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7879168-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7879168/providers/microsoft.monitor/accounts/hcps-j7879168", namespace="ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fd1ced12-4634-450b-be94-fe821eb628d0"
  Description: Pod ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 18:15	PR #7168	2103549055647879168

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-25T03:25:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7908608-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7908608/providers/microsoft.monitor/accounts/services-j7908608", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dq66f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-25T03:26:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7908608-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7908608/providers/microsoft.monitor/accounts/services-j7908608", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dq66f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:07	PR #7148	2103305281587908608

alert [svc] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [svc] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-25T03:35:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7908608-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7908608/providers/microsoft.monitor/accounts/services-j7908608", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dq66f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-25T04:01:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7908608-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7908608/providers/microsoft.monitor/accounts/services-j7908608", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dq66f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:07	PR #7148	2103305281587908608

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-25T03:00:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7908608-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-78648d9c8d-tkfsj", region="westus3", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-78648d9c8d-tkfsj has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:07	PR #7148	2103305281587908608

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:136]: failed to get admin REST config for cluster "cluster-zstream-4-21-kz5vz4"
Unexpected error:
    <*fmt.wrapError | 0xc0010728c0>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-kz5vz4" in resourcegroup="rg-zstream-upgrade-4-21-xtgkx9mrrcpz" to finish getting creds: GET https://rp.j0275968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/773f5d4a-cd7a-4e25-ac56-58130cbed2d2
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/773f5d4a-cd7a-4e25-ac56-58130cbed2d2' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/773f5d4a-cd7a-4e25-ac56-58130cbed2d2"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:136]: failed to get admin REST config for cluster "cluster-zstream-4-21-kz5vz4"
Unexpected error:
    <*fmt.wrapError | 0xc0010728c0>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-kz5vz4" in resourcegroup="rg-zstream-upgrade-4-21-xtgkx9mrrcpz" to finish getting creds: GET https://rp.j0275968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/773f5d4a-cd7a-4e25-ac56-58130cbed2d2
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/773f5d4a-cd7a-4e25-ac56-58130cbed2d2' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/773f5d4a-cd7a-4e25-ac56-58130cbed2d2"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 11:48	PR #6577	2103451563560275968

alert [svc] MaestroAgentMetricsProxyDown fired

full failure pattern:

alert [svc] MaestroAgentMetricsProxyDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-25T11:11:32Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j7570432-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 10:20	PR #7164	2103429520567570432

client rate limiter Wait returned an error: context deadline exceeded

full failure pattern:

client rate limiter Wait returned an error: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:286]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000d1f800>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:286]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000d1f800>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 20:17	PR #7169	2103579630278545408

ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:97]: failed to create customer resources for basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c324a0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-rjv7q9" in resourcegroup="rg-cluster-nsg-subnet-reuse-9crmxlzr24l9" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-9crmxlzr24l9/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rjv7q9/operationStatuses/08584112672274846775
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xafodxhow6npw' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:97]: failed to create customer resources for basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c324a0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-rjv7q9" in resourcegroup="rg-cluster-nsg-subnet-reuse-9crmxlzr24l9" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-9crmxlzr24l9/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rjv7q9/operationStatuses/08584112672274846775
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xafodxhow6npw' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 11:48	PR #6577	2103451563560275968

ERROR CODE: InternalServerError; detail code OperationNotAllowed; detail message Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-<id>' since...

full failure pattern:

ERROR CODE: InternalServerError; detail code OperationNotAllowed; detail message Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-<id>' since another operation is in progress.; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00085dbd8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-566qcntr7gdn" to finish creating: GET https://rp.j4621824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47438056-285b-41f2-ab31-9d3620f43054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47438056-285b-41f2-ab31-9d3620f43054",
      "name": "47438056-285b-41f2-ab31-9d3620f43054",
      "status": "Failed",
      "startTime": "2026-09-25T13:28:05.551349476Z",
      "endTime": "2026-09-25T13:47:09.734640501Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-h9wwd-6cst6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-566qcntr7gdn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-h9wwd-6cst6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-h9wwd-6cst6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-566qcntr7gdn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-h9wwd-6cst6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-h9wwd-6cst6": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-h9wwd-6cst6" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-566qcntr7gdn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-h9wwd-6cst6/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-h9wwd-6cst6' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00085dbd8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-566qcntr7gdn" to finish creating: GET https://rp.j4621824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47438056-285b-41f2-ab31-9d3620f43054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47438056-285b-41f2-ab31-9d3620f43054",
      "name": "47438056-285b-41f2-ab31-9d3620f43054",
      "status": "Failed",
      "startTime": "2026-09-25T13:28:05.551349476Z",
      "endTime": "2026-09-25T13:47:09.734640501Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-h9wwd-6cst6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-566qcntr7gdn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-h9wwd-6cst6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-h9wwd-6cst6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-566qcntr7gdn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-h9wwd-6cst6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-h9wwd-6cst6": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-h9wwd-6cst6" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-566qcntr7gdn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-h9wwd-6cst6/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-h9wwd-6cst6' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 12:26	PR #7163	2103461242604621824

ERROR: Please run 'az login' to setup account.

full failure pattern:

ERROR: Please run 'az login' to setup account.

Signal:

provision

none

Full failure examples (1)

time=2026-09-25T09:33:51.597Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 description="Step swift-vnet\n  Kind: Shell\n  Command: ./swift-vnet.sh\n"
time=2026-09-25T09:33:52.825Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: Launching container group swift-vnet-aks-net as globalMSI...\nERROR: Please run 'az login' to setup account.\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/swift-vnet (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 09:08	PR #7164	2103411413878837248

VerifyImagePulled(acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal) timed out after 3m0.006s: expected image from acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal in namespace acr...

full failure pattern:

VerifyImagePulled(acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal) timed out after 3m0.006s: expected image from acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal in namespace acr-pull-test: image pull errors:

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:397]: failed to pull image after updating containerRegistry to second MI
Unexpected error:
    <*fmt.wrapError | 0xc000b92200>: 
    VerifyImagePulled(acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal) timed out after 3m0.006s: expected image from acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal in namespace acr-pull-test: image pull errors:
      - pod acr-pull-test-after-update container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: eefe0445-30e7-478a-8283-14278233fefa; artifact err: get manifest: build image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: c9d8d031-808a-4e35-9c9c-bff18e860fbb
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:397]: failed to pull image after updating containerRegistry to second MI
Unexpected error:
    <*fmt.wrapError | 0xc000b92200>: 
    VerifyImagePulled(acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal) timed out after 3m0.006s: expected image from acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal in namespace acr-pull-test: image pull errors:
      - pod acr-pull-test-after-update container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullcreatefddp5k2l7rsk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: eefe0445-30e7-478a-8283-14278233fefa; artifact err: get manifest: build image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: c9d8d031-808a-4e35-9c9c-bff18e860fbb
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 20:17	PR #7169	2103579630278545408

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...

full failure pattern:

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc00145e060>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-xz45t.apps.aro.x9e6g8g9i5v5d8w.8r9s.j1359744.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-xz45t.apps.aro.x9e6g8g9i5v5d8w.8r9s.j1359744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xz45t.apps.aro.x9e6g8g9i5v5d8w.8r9s.j1359744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc00145e060>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-xz45t.apps.aro.x9e6g8g9i5v5d8w.8r9s.j1359744.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-xz45t.apps.aro.x9e6g8g9i5v5d8w.8r9s.j1359744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xz45t.apps.aro.x9e6g8g9i5v5d8w.8r9s.j1359744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 14:02	PR #7165	2103485306291359744

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no s...

full failure pattern:

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:209]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001afc030>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-kzc9z.apps.aro.cilium-cluster.oaay.j3069568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-kzc9z.apps.aro.cilium-cluster.oaay.j3069568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kzc9z.apps.aro.cilium-cluster.oaay.j3069568.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:209]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001afc030>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-kzc9z.apps.aro.cilium-cluster.oaay.j3069568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-kzc9z.apps.aro.cilium-cluster.oaay.j3069568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kzc9z.apps.aro.cilium-cluster.oaay.j3069568.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 19:17	PR #7176	2103564467513069568

alert [svc] MaestroEventStuckUnreconciled fired

full failure pattern:

alert [svc] MaestroEventStuckUnreconciled fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T16:32:06Z
  Ended:       2026-09-25T16:39:07Z
  Severity:    Sev3
  Labels:      alertname="MaestroEventStuckUnreconciled", component="maestro", region="westus3", severity="warning"
  Description: A Maestro event has been unreconciled for 6m 6s. This likely indicates a lost Postgres NOTIFY message. The event will not be processed until the periodic sync runs (default 10 hours).

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroEventStuckUnreconciled does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 15:23	PR #7176	2103505655406006272

alert [svc] MaestroSpecControllerReconcileErrors fired

full failure pattern:

alert [svc] MaestroSpecControllerReconcileErrors fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T16:36:07Z
  Ended:       2026-09-25T16:58:05Z
  Severity:    Sev3
  Labels:      alertname="MaestroSpecControllerReconcileErrors", component="maestro", region="westus3", severity="warning"
  Description: Maestro spec controller reconcile error rate is above 10% for the last 10 minutes. Resources may not be reaching management clusters. Current value: 20.38%.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroSpecControllerReconcileErrors does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 15:23	PR #7176	2103505655406006272

Environment: INT

Failure patterns: 3 · Failures matched: 6 · Total runs: 7 · Runs affected: 4

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machine...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute

Signal:

e2e

DEV, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000599de0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-kcj4l5" in resourcegroup="rg-np-skip-minor-kcj4l5-45s8nhm4k78b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d",
      "name": "da564fd3-aea3-4a41-b171-3969372d023d",
      "status": "Failed",
      "startTime": "2026-09-25T16:39:04.653915252Z",
      "endTime": "2026-09-25T16:58:09.716908828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000599de0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-kcj4l5" in resourcegroup="rg-np-skip-minor-kcj4l5-45s8nhm4k78b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d",
      "name": "da564fd3-aea3-4a41-b171-3969372d023d",
      "status": "Failed",
      "startTime": "2026-09-25T16:39:04.653915252Z",
      "endTime": "2026-09-25T16:58:09.716908828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00075fc80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-dnhw57q2zhsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "name": "9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "status": "Failed",
      "startTime": "2026-09-25T08:30:53.138668141Z",
      "endTime": "2026-09-25T08:49:53.966422643Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00075fc80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-dnhw57q2zhsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "name": "9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "status": "Failed",
      "startTime": "2026-09-25T08:30:53.138668141Z",
      "endTime": "2026-09-25T08:49:53.966422643Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010c02c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2mw49n77wnb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "name": "e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "status": "Failed",
      "startTime": "2026-09-25T08:27:10.94071903Z",
      "endTime": "2026-09-25T08:46:13.959695236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010c02c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2mw49n77wnb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "name": "e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "status": "Failed",
      "startTime": "2026-09-25T08:27:10.94071903Z",
      "endTime": "2026-09-25T08:46:13.959695236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-25 15:01	n/a	2103500347635929088
2026-09-25 08:13	n/a	2103397522356572160
2026-09-25 02:01	n/a	2103303905172525056

ERROR CODE: Conflict; detail message Cannot update resource while resource is "provisioning"; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: Conflict; detail message Cannot update resource while resource is "provisioning"; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx"
Unexpected error:
    <*fmt.wrapError | 0xc000c10900>: 
    failed to create NodePool np-pt5qmx: failed starting nodepool creation "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx" in resourcegroup="rg-candidate-5-0-9clwz8rf6599": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx"
Unexpected error:
    <*fmt.wrapError | 0xc000c10900>: 
    failed to create NodePool np-pt5qmx: failed starting nodepool creation "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx" in resourcegroup="rg-candidate-5-0-9clwz8rf6599": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 05:49	n/a	2103361291237199872

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000f18e80>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-p9kv2wz66ng2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1",
      "name": "82d3308e-222c-40d8-a1be-8c057c26bab1",
      "status": "Failed",
      "startTime": "2026-09-25T15:45:43.486812846Z",
      "endTime": "2026-09-25T16:14:49.560161271Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000f18e80>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-p9kv2wz66ng2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1",
      "name": "82d3308e-222c-40d8-a1be-8c057c26bab1",
      "status": "Failed",
      "startTime": "2026-09-25T15:45:43.486812846Z",
      "endTime": "2026-09-25T16:14:49.560161271Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 15:01	n/a	2103500347635929088

Environment: PROD

Failure patterns: 8 · Failures matched: 17 · Total runs: 2 · Runs affected: 2

Failure pattern

Failed at

Trend

Also in

storage account log verification failed

full failure pattern:

storage account log verification failed

Signal:

e2e

STG

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxlj58f6 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-5tckvstjpzw8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0011fb4a0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxlj58f6 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-5tckvstjpzw8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0011fb4a0>: 
    storage account log verification failed
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxsjrhvt and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-722np9bhtr46/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0007b7cd0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxsjrhvt and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-722np9bhtr46/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0007b7cd0>: 
    storage account log verification failed
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620630963294208
2026-09-25 02:00	n/a	2103303533200674816

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc00117a060>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-8xvvbqc2kcgw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc00117a060>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-8xvvbqc2kcgw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0011d8600>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8l7wjgzcr5kh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0011d8600>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8l7wjgzcr5kh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0013e4db0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-t66lgcklqd6q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0013e4db0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-t66lgcklqd6q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620630963294208

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc001025cc0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-zlflh9" in resourcegroup="rg-np-skip-minor-zlflh9-nz2rc2k8lntd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zlflh9-nz2rc2k8lntd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zlflh9/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc001025cc0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-zlflh9" in resourcegroup="rg-np-skip-minor-zlflh9-nz2rc2k8lntd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zlflh9-nz2rc2k8lntd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zlflh9/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc001419e20>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lp7f8j" resourcegroup="rg-np-dg-minor-lp7f8j-kb9jlc2jfsr2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lp7f8j-kb9jlc2jfsr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lp7f8j/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc001419e20>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lp7f8j" resourcegroup="rg-np-dg-minor-lp7f8j-kb9jlc2jfsr2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lp7f8j-kb9jlc2jfsr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lp7f8j/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-sf8c2g" for cluster "cluster-nightly-4-20-sf8c2g"
Unexpected error:
    <*fmt.wrapError | 0xc000949500>: 
    failed to create NodePool np-sf8c2g: failed starting nodepool creation "np-sf8c2g" for cluster "cluster-nightly-4-20-sf8c2g" in resourcegroup="rg-nightly-4-20-7swt4w2kzhfs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-7swt4w2kzhfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-sf8c2g/nodePools/np-sf8c2g
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-sf8c2g" for cluster "cluster-nightly-4-20-sf8c2g"
Unexpected error:
    <*fmt.wrapError | 0xc000949500>: 
    failed to create NodePool np-sf8c2g: failed starting nodepool creation "np-sf8c2g" for cluster "cluster-nightly-4-20-sf8c2g" in resourcegroup="rg-nightly-4-20-7swt4w2kzhfs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-7swt4w2kzhfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-sf8c2g/nodePools/np-sf8c2g
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620630963294208

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc0008ef760>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-jbgszp" in resourcegroup="rg-np-version-upgrade-jbgszp-szxzcp5vvtll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/767328b3-e719-4dce-a9f8-d13c4b4802eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/767328b3-e719-4dce-a9f8-d13c4b4802eb",
      "name": "767328b3-e719-4dce-a9f8-d13c4b4802eb",
      "status": "Failed",
      "startTime": "2026-09-25T03:20:28.674763184Z",
      "endTime": "2026-09-25T03:39:37.254782397Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc0008ef760>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-jbgszp" in resourcegroup="rg-np-version-upgrade-jbgszp-szxzcp5vvtll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/767328b3-e719-4dce-a9f8-d13c4b4802eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/767328b3-e719-4dce-a9f8-d13c4b4802eb",
      "name": "767328b3-e719-4dce-a9f8-d13c4b4802eb",
      "status": "Failed",
      "startTime": "2026-09-25T03:20:28.674763184Z",
      "endTime": "2026-09-25T03:39:37.254782397Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00013d940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-pv4shddpb7qx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/979880cc-8a28-44f5-8525-bb6028b5328e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/979880cc-8a28-44f5-8525-bb6028b5328e",
      "name": "979880cc-8a28-44f5-8525-bb6028b5328e",
      "status": "Failed",
      "startTime": "2026-09-25T02:12:08.565557217Z",
      "endTime": "2026-09-25T02:31:16.095644453Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00013d940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-pv4shddpb7qx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/979880cc-8a28-44f5-8525-bb6028b5328e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/979880cc-8a28-44f5-8525-bb6028b5328e",
      "name": "979880cc-8a28-44f5-8525-bb6028b5328e",
      "status": "Failed",
      "startTime": "2026-09-25T02:12:08.565557217Z",
      "endTime": "2026-09-25T02:31:16.095644453Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:00	n/a	2103303533200674816

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

STG

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000bc5170>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000bc5170>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620630963294208

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-7q8vqcfshfst/cluster-nightly-5-1-htcp7l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000acaa60>: 
    failed to create HCP cluster cluster-nightly-5-1-htcp7l: failed waiting for cluster="cluster-nightly-5-1-htcp7l" in resourcegroup="rg-nightly-5-1-7q8vqcfshfst" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "name": "3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "status": "Failed",
      "startTime": "2026-09-25T23:32:03.179098884Z",
      "endTime": "2026-09-26T00:01:10.002965076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-7q8vqcfshfst/cluster-nightly-5-1-htcp7l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000acaa60>: 
    failed to create HCP cluster cluster-nightly-5-1-htcp7l: failed waiting for cluster="cluster-nightly-5-1-htcp7l" in resourcegroup="rg-nightly-5-1-7q8vqcfshfst" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "name": "3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "status": "Failed",
      "startTime": "2026-09-25T23:32:03.179098884Z",
      "endTime": "2026-09-26T00:01:10.002965076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620630963294208

ERROR CODE: InternalServerError; detail code OperationNotAllowed; detail message Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-<id>'...

full failure pattern:

ERROR CODE: InternalServerError; detail code OperationNotAllowed; detail message Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-<id>' since another operation is in progress.; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000ff01b0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-d8dncw5q6xr2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735",
      "name": "4cccec67-451f-41c1-957a-1453bb0ac735",
      "status": "Failed",
      "startTime": "2026-09-25T02:15:20.351974211Z",
      "endTime": "2026-09-25T02:34:26.038545442Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-44b92-qd24g": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-44b92-qd24g" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-d8dncw5q6xr2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-44b92-qd24g/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-qd24g' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000ff01b0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-d8dncw5q6xr2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735",
      "name": "4cccec67-451f-41c1-957a-1453bb0ac735",
      "status": "Failed",
      "startTime": "2026-09-25T02:15:20.351974211Z",
      "endTime": "2026-09-25T02:34:26.038545442Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-44b92-qd24g": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-44b92-qd24g" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-d8dncw5q6xr2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-44b92-qd24g/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-qd24g' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 02:00	n/a	2103303533200674816

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00078b060>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gb52bkffcwft" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/db1ab9c2-9577-48f8-9f15-b41d89774bb7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/db1ab9c2-9577-48f8-9f15-b41d89774bb7",
      "name": "db1ab9c2-9577-48f8-9f15-b41d89774bb7",
      "status": "Failed",
      "startTime": "2026-09-25T23:59:44.019265481Z",
      "endTime": "2026-09-26T00:18:50.149741392Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00078b060>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gb52bkffcwft" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/db1ab9c2-9577-48f8-9f15-b41d89774bb7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/db1ab9c2-9577-48f8-9f15-b41d89774bb7",
      "name": "db1ab9c2-9577-48f8-9f15-b41d89774bb7",
      "status": "Failed",
      "startTime": "2026-09-25T23:59:44.019265481Z",
      "endTime": "2026-09-26T00:18:50.149741392Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to delete a nodepool	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620630963294208

Environment: STG

Failure patterns: 8 · Failures matched: 18 · Total runs: 5 · Runs affected: 5

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machine...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute

Signal:

e2e

DEV, INT

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000feb180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-jmvth8gmbtdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13",
      "name": "c689250c-de76-4270-9b21-5b67d697be13",
      "status": "Failed",
      "startTime": "2026-09-25T16:52:06.978905427Z",
      "endTime": "2026-09-25T17:11:15.434963979Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000feb180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-jmvth8gmbtdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13",
      "name": "c689250c-de76-4270-9b21-5b67d697be13",
      "status": "Failed",
      "startTime": "2026-09-25T16:52:06.978905427Z",
      "endTime": "2026-09-25T17:11:15.434963979Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000e30fe0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-zw4cd4" in resourcegroup="rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012",
      "name": "995a0744-273b-428e-a7c0-5da145da2012",
      "status": "Failed",
      "startTime": "2026-09-25T17:47:45.849510742Z",
      "endTime": "2026-09-25T18:06:46.935940984Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000e30fe0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-zw4cd4" in resourcegroup="rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012",
      "name": "995a0744-273b-428e-a7c0-5da145da2012",
      "status": "Failed",
      "startTime": "2026-09-25T17:47:45.849510742Z",
      "endTime": "2026-09-25T18:06:46.935940984Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001140700>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-98gh8s6zjjg9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c466aed6-2164-42ab-a3c4-c5d9337db2b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c466aed6-2164-42ab-a3c4-c5d9337db2b5",
      "name": "c466aed6-2164-42ab-a3c4-c5d9337db2b5",
      "status": "Failed",
      "startTime": "2026-09-25T05:20:30.850301491Z",
      "endTime": "2026-09-25T05:39:34.701925259Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-98gh8s6zjjg9--managed/providers/Microsoft.Compute/virtualMachines/v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-98gh8s6zjjg9--managed/providers/Microsoft.Compute/virtualMachines/v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001140700>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-98gh8s6zjjg9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c466aed6-2164-42ab-a3c4-c5d9337db2b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c466aed6-2164-42ab-a3c4-c5d9337db2b5",
      "name": "c466aed6-2164-42ab-a3c4-c5d9337db2b5",
      "status": "Failed",
      "startTime": "2026-09-25T05:20:30.850301491Z",
      "endTime": "2026-09-25T05:39:34.701925259Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-98gh8s6zjjg9--managed/providers/Microsoft.Compute/virtualMachines/v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-98gh8s6zjjg9--managed/providers/Microsoft.Compute/virtualMachines/v5v6d2f0y9a3z6e-np-1-r5nkw-flxjm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	2
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 16:19	n/a	2103519896854859776
2026-09-25 04:43	n/a	2103344681369735168

storage account log verification failed

full failure pattern:

storage account log verification failed

Signal:

e2e

PROD

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxn248k7 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-hwjzhdrfqdml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000973dd0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxn248k7 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-hwjzhdrfqdml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000973dd0>: 
    storage account log verification failed
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4lqltr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-gvcrd9xg4vzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0008ace00>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4lqltr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-gvcrd9xg4vzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0008ace00>: 
    storage account log verification failed
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-25 04:43	n/a	2103344681369735168
2026-09-25 02:00	n/a	2103303533246812160

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00088f9e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7kjs5ps99kbv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00088f9e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7kjs5ps99kbv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0010fe8a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-srk662g7xx9b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0010fe8a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-srk662g7xx9b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0020da1e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-z8zt66k27vt9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0020da1e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-z8zt66k27vt9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620631026208768

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dt4hdm" for cluster "cluster-nightly-4-20-dt4hdm"
Unexpected error:
    <*fmt.wrapError | 0xc000600e20>: 
    failed to create NodePool np-dt4hdm: failed starting nodepool creation "np-dt4hdm" for cluster "cluster-nightly-4-20-dt4hdm" in resourcegroup="rg-nightly-4-20-mbwzhwdx5487": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-mbwzhwdx5487/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-dt4hdm/nodePools/np-dt4hdm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dt4hdm" for cluster "cluster-nightly-4-20-dt4hdm"
Unexpected error:
    <*fmt.wrapError | 0xc000600e20>: 
    failed to create NodePool np-dt4hdm: failed starting nodepool creation "np-dt4hdm" for cluster "cluster-nightly-4-20-dt4hdm" in resourcegroup="rg-nightly-4-20-mbwzhwdx5487": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-mbwzhwdx5487/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-dt4hdm/nodePools/np-dt4hdm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000a169e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-8wcm2h" in resourcegroup="rg-np-version-upgrade-8wcm2h-qhfg24hmjm4w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8wcm2h-qhfg24hmjm4w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8wcm2h/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000a169e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-8wcm2h" in resourcegroup="rg-np-version-upgrade-8wcm2h-qhfg24hmjm4w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-8wcm2h-qhfg24hmjm4w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-8wcm2h/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc001659e40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-ftxd2m" resourcegroup="rg-np-dg-minor-ftxd2m-fl7b6d52hcz6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-ftxd2m-fl7b6d52hcz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-ftxd2m/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc001659e40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-ftxd2m" resourcegroup="rg-np-dg-minor-ftxd2m-fl7b6d52hcz6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-ftxd2m-fl7b6d52hcz6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-ftxd2m/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620631026208768

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc001409500>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc001409500>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-24-142420, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620631026208768

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster servi...

full failure pattern:

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-tklhx4zzdx6h/cluster-nightly-5-1-vjz6jm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e262a0>: 
    failed to create HCP cluster cluster-nightly-5-1-vjz6jm: failed waiting for cluster="cluster-nightly-5-1-vjz6jm" in resourcegroup="rg-nightly-5-1-tklhx4zzdx6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e",
      "name": "5f9063e9-f978-4377-b5d9-c9222622088e",
      "status": "Failed",
      "startTime": "2026-09-26T00:24:56.11201607Z",
      "endTime": "2026-09-26T00:54:00.860881838Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-tklhx4zzdx6h/cluster-nightly-5-1-vjz6jm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e262a0>: 
    failed to create HCP cluster cluster-nightly-5-1-vjz6jm: failed waiting for cluster="cluster-nightly-5-1-vjz6jm" in resourcegroup="rg-nightly-5-1-tklhx4zzdx6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e",
      "name": "5f9063e9-f978-4377-b5d9-c9222622088e",
      "status": "Failed",
      "startTime": "2026-09-26T00:24:56.11201607Z",
      "endTime": "2026-09-26T00:54:00.860881838Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 23:00	n/a	2103620631026208768

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want >= 2 (autoscaling min); node pool condition AllNodesHealthy is ...

full failure pattern:

ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want >= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-<id> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-<id> to exist; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0015ba9f0>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-2rtwrg4pjxl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714",
      "name": "88abacf9-a5e0-4248-b32c-83d6cd062714",
      "status": "Failed",
      "startTime": "2026-09-25T22:23:02.156910064Z",
      "endTime": "2026-09-25T22:42:06.143243532Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2klmb-6glqn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2klmb-6glqn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0015ba9f0>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-2rtwrg4pjxl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714",
      "name": "88abacf9-a5e0-4248-b32c-83d6cd062714",
      "status": "Failed",
      "startTime": "2026-09-25T22:23:02.156910064Z",
      "endTime": "2026-09-25T22:42:06.143243532Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2klmb-6glqn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2klmb-6glqn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 21:51	n/a	2103603313239920640

Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again

full failure pattern:

Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:214]: failed to cordon node d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6
Unexpected error:
    <*errors.StatusError | 0xc000b17180>: 
    Operation cannot be fulfilled on nodes "d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:214]: failed to cordon node d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6
Unexpected error:
    <*errors.StatusError | 0xc000b17180>: 
    Operation cannot be fulfilled on nodes "d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-25 04:43	n/a	2103344681369735168